hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3
Subsystems (2)
v5 cover hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3
Hi,
Changes from v4:
https://lore.kernel.org/qemu-devel/[email protected]/
- Addressed v4 feedback and picked up Reviewed-by tags. Thanks!
- Improved comments to specify optional/mandatory for CMDQ ops(patch#6)
- Introduced writel_mmio for 4-byte and writell_mmio for 8-byte(patch #14)
- Separated vcmdq Page0 and Page1 access helpers(patches #15, #16)
- Ignore VCMDQ BASE writes if the VCMDQ is already enabled(patch #16)
- Modified address_space_is_ram helper to address_space_range_is_ram() to cover the entire range(patch #18)
- Allocate HW QUEUE once all pre-conditions are met(BASE write, CMDQ_ALLOC_MAP, and CMDQV / VINTF enabled) - patch #19
- Sync the VCMDQ Page0 register cache when mmap'd VINTF page0 is available(patch #20)
- Introduced memory_region_skip_iommu_map()/_set_skip_iommu_map() accessors(patch 21).
- Moved the renamed helper qemu_ram_backend_pagesize_min() to system/physmem.c (patch #26)
- Updated commit log with DSDT/IORT AML descriptions(patch #30).
- Based on top of Nathan's "Resolve AUTO properties" v3 series [0].
Please find the complete branch here:
https://github.com/shamiali2008/qemu-master/tree/master-vcmdq-v5-ext
Sanity tested on NVIDIA Grace. Further testing in progress.
Feedback and testing are very welcome.
Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/
---
Background(from RFCv1):
https://lore.kernel.org/qemu-devel/[email protected]/
Thanks to Nicolin for the initial patches and testing on which this is based.
Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.
Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)
Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.
VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].
---
Nicolin Chen (15):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Implement CMDQV init hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT
Shameer Kolothum (17):
system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq hw/arm/smmuv3-accel: Introduce CMDQV ops interface hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle hw/arm/virt: Use stored SMMUv3 device list for IORT build hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus system/physmem: Add address_space_range_is_ram() helper hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
memory: Allow RAM device regions to skip IOMMU mapping hw/arm/smmuv3-accel: Introduce common helper for veventq read hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors system/physmem: Add qemu_ram_backend_pagesize_min() helper hw/arm/smmuv3: Add per-device identifier property hw/arm/smmuv3-accel: Introduce helper to query CMDQV type hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active hw/arm/smmuv3: Add cmdqv property for SMMUv3 device
hw/arm/smmuv3-accel.h | 49 ++ hw/arm/tegra241-cmdqv.h | 381 +++++++++++++ include/hw/arm/smmuv3.h | 3 + include/hw/arm/virt.h | 1 + include/system/iommufd.h | 17 +- include/system/memory.h | 34 ++ include/system/ramblock.h | 1 + backends/iommufd.c | 64 +++ hw/arm/smmuv3-accel-stubs.c | 12 + hw/arm/smmuv3-accel.c | 196 +++++-- hw/arm/smmuv3.c | 10 + hw/arm/tegra241-cmdqv-stubs.c | 16 + hw/arm/tegra241-cmdqv.c | 1007 +++++++++++++++++++++++++++++++++ hw/arm/virt-acpi-build.c | 127 +++-- hw/arm/virt.c | 37 ++ hw/vfio/iommufd.c | 4 +- hw/vfio/listener.c | 4 + system/memory.c | 10 + system/physmem.c | 48 ++ backends/trace-events | 4 +- hw/arm/Kconfig | 5 + hw/arm/meson.build | 2 + hw/arm/trace-events | 11 + qemu-options.hx | 8 + 24 files changed, 1971 insertions(+), 80 deletions(-) create mode 100644 hw/arm/tegra241-cmdqv.h create mode 100644 hw/arm/tegra241-cmdqv-stubs.c create mode 100644 hw/arm/tegra241-cmdqv.c
Signature · 1 line
2.43.0
PATCH v5 1/32 backends/iommufd: Update iommufd_backend_get_device_info
From: Nicolin Chen <[email protected]>
The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass a type value explicitly initialised to zero (IOMMU_HW_INFO_TYPE_DEFAULT) when no specific type is requested.
--- backends/iommufd.c | 7 +++++++ hw/arm/smmuv3-accel.c | 2 +- hw/vfio/iommufd.c | 4 ++-- 3 files changed, 10 insertions(+), 3 deletions(-)
backends/iommufd.c +7 -0
@@ -390,16 +390,23 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be,return true; } +/* +* @type can carry a desired HW info type defined in the uapi headers. If caller +* doesn't have one, indicating it wants the default type, then @type should be +* zeroed (i.e. IOMMU_HW_INFO_TYPE_DEFAULT). +*/ bool iommufd_backend_get_device_info (IOMMUFDBackend *be,uint32_t devid,uint32_t *type,void *data,uint32_t len,uint64_t *caps,uint8_t *max_pasid_log2,Error **errp) {struct iommu_hw_info info = { + .flags = (*type) ?IOMMU_HW_INFO_FLAG_INPUT_TYPE :0 , .size =sizeof (info), .dev_id = devid, .data_len = len, .data_uptr = (uintptr_t )data, + .in_data_type = *type, };if (ioctl (be->fd,IOMMU_GET_HW_INFO , &info)) {
hw/arm/smmuv3-accel.c +1 -1
@@ -176,7 +176,7 @@ smmuv3_accel_hw_compatible(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, Error **errp) { struct iommu_hw_info_arm_smmuv3 info; - uint32_t data_type; + uint32_t data_type = IOMMU_HW_INFO_TYPE_DEFAULT; uint64_t caps; if (!iommufd_backend_get_device_info(hiodi->iommufd, hiodi->devid,
hw/vfio/iommufd.c +2 -2
@@ -353,7 +353,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,IOMMUFDBackend *iommufd = vbasedev->iommufd;VFIOContainer *bcontainer =VFIO_IOMMU (container);bool viommu_nesting, viommu_nesting_dirty; -uint32_t type, flags =0 ; +uint32_t type =IOMMU_HW_INFO_TYPE_DEFAULT , flags =0 ;uint64_t hw_caps;VendorCaps caps;VFIOIOASHwpt *hwpt;
@@ -948,7 +948,7 @@ static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque,HostIOMMUDeviceIOMMUFD *hiodi;HostIOMMUDeviceCaps *caps = &hiod->caps;VendorCaps *vendor_caps = &caps->vendor_caps; -enum iommu_hw_info_type type; +uint32_t type =IOMMU_HW_INFO_TYPE_DEFAULT ;uint8_t max_pasid_log2;uint64_t hw_caps;
Signature · 1 line
2.43.0
PATCH v5 2/32 backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
The updated IOMMUFD VIOMMU_ALLOC uAPI allows userspace to provide a data buffer when creating a vIOMMU (e.g. for Tegra241 CMDQV). Extend iommufd_backend_alloc_viommu() to pass a user pointer and size to the kernel.
Update the caller accordingly.
--- include/system/iommufd.h | 1 + backends/iommufd.c | 4 ++++ hw/arm/smmuv3-accel.c | 4 ++-- backends/trace-events | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-)
include/system/iommufd.h +1 -0
@@ -89,6 +89,7 @@ bool iommufd_backend_alloc_hwpt(IOMMUFDBackend *be, uint32_t dev_id, Error **errp);bool iommufd_backend_alloc_viommu (IOMMUFDBackend *be,uint32_t dev_id,uint32_t viommu_type,uint32_t hwpt_id, +void *data_ptr,uint32_t data_len,uint32_t *out_hwpt,Error **errp); bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
backends/iommufd.c +4 -0
@@ -463,6 +463,7 @@ bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id,bool iommufd_backend_alloc_viommu (IOMMUFDBackend *be,uint32_t dev_id,uint32_t viommu_type,uint32_t hwpt_id, +void *data_ptr,uint32_t data_len,uint32_t *out_viommu_id,Error **errp) {int ret;
@@ -471,11 +472,14 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id, .type = viommu_type, .dev_id = dev_id, .hwpt_id = hwpt_id, + .data_len = data_len, + .data_uptr = (uintptr_t)data_ptr, }; ret =ioctl (be->fd,IOMMU_VIOMMU_ALLOC , &alloc_viommu);trace_iommufd_backend_alloc_viommu (be->fd, dev_id, viommu_type, hwpt_id, + (uintptr_t )data_ptr, data_len, alloc_viommu.out_viommu_id, ret);if (ret) {error_setg_errno (errp, errno,"IOMMU_VIOMMU_ALLOC failed" );
hw/arm/smmuv3-accel.c +2 -2
@@ -582,8 +582,8 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,IOMMUFDViommu *viommu;if (!iommufd_backend_alloc_viommu (hiodi->iommufd, hiodi->devid, -IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , - s2_hwpt_id, &viommu_id, errp)) { +IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , s2_hwpt_id, +NULL ,0 , &viommu_id, errp)) {return false; }
backends/trace-events +1 -1
@@ -21,7 +21,7 @@ iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (% iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%u enable=%d (%d)" iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, uint64_t size, uint64_t flags, uint64_t page_size, int ret) " iommufd=%d hwpt=%u iova=0x%"PRIx64" size=0x%"PRIx64" flags=0x%"PRIx64" page_size=0x%"PRIx64" (%d)" iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, uint32_t entry_len, uint32_t entry_num, uint32_t done_num, uint64_t data_ptr, int ret) " iommufd=%d id=%u data_type=%u entry_len=%u entry_num=%u done_num=%u data_ptr=0x%"PRIx64" (%d)" -iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)" +iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)" iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)" iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
Signature · 1 line
2.43.0
PATCH v5 3/32 backends/iommufd: Introduce iommufd_backend_alloc_hw_queue
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add a helper to allocate an iommufd backed HW queue for a vIOMMU.
While at it, define a struct IOMMUFDHWqueue for use by vendor implementations.
--- include/system/iommufd.h | 11 +++++++++++ backends/iommufd.c | 31 +++++++++++++++++++++++++++++++ backends/trace-events | 1 + 3 files changed, 43 insertions(+)
include/system/iommufd.h +11 -0
@@ -65,6 +65,12 @@ typedef struct IOMMUFDVeventq {bool event_start;/* True after first valid event; cleared on overflow */ } IOMMUFDVeventq; +/* HW queue object for a vIOMMU-specific HW-accelerated queue */ +typedef struct IOMMUFDHWqueue { +IOMMUFDViommu *viommu; +uint32_t hw_queue_id; +}IOMMUFDHWqueue ; +bool iommufd_backend_connect (IOMMUFDBackend *be,Error **errp);void iommufd_backend_disconnect (IOMMUFDBackend *be);
@@ -101,6 +107,11 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,uint32_t *out_veventq_id, uint32_t *out_veventq_fd, Error **errp); +bool iommufd_backend_alloc_hw_queue (IOMMUFDBackend *be,uint32_t viommu_id, +uint32_t queue_type,uint32_t index, +uint64_t addr,uint64_t length, +uint32_t *out_hw_queue_id,Error **errp); +bool iommufd_backend_set_dirty_tracking (IOMMUFDBackend *be,uint32_t hwpt_id,bool start,Error **errp); bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +31 -0
@@ -549,6 +549,37 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,return true; } +bool iommufd_backend_alloc_hw_queue (IOMMUFDBackend *be,uint32_t viommu_id, +uint32_t queue_type,uint32_t index, +uint64_t addr,uint64_t length, +uint32_t *out_hw_queue_id,Error **errp) +{ +int ret; +struct iommu_hw_queue_alloc alloc_hw_queue = { + .size =sizeof (alloc_hw_queue), + .flags =0 , + .viommu_id = viommu_id, + .type = queue_type, + .index = index, + .nesting_parent_iova = addr, + .length = length, + }; + + ret =ioctl (be->fd,IOMMU_HW_QUEUE_ALLOC , &alloc_hw_queue); + +trace_iommufd_backend_alloc_hw_queue (be->fd, viommu_id, queue_type, + index, addr, length, + alloc_hw_queue.out_hw_queue_id, ret); +if (ret) { +error_setg_errno (errp, errno,"IOMMU_HW_QUEUE_ALLOC failed" ); +return false; + } + +g_assert (out_hw_queue_id); + *out_hw_queue_id = alloc_hw_queue.out_hw_queue_id; +return true; +} +bool host_iommu_device_iommufd_attach_hwpt (HostIOMMUDeviceIOMMUFD *hiodi,uint32_t hwpt_id,Error **errp) {
backends/trace-events +1 -0
@@ -24,6 +24,7 @@ iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)" iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)" iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)" +iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)" # igvm-cfg.c igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
PATCH v5 4/32 backends/iommufd: Introduce iommufd_backend_viommu_mmap
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add a backend helper to mmap hardware MMIO regions exposed via iommufd for a vIOMMU instance. This allows user space to access HW-accelerated MMIO pages provided by the vIOMMU.
The caller is responsible for unmapping the returned region.
--- include/system/iommufd.h | 4 ++++ backends/iommufd.c | 22 ++++++++++++++++++++++ backends/trace-events | 1 + 3 files changed, 27 insertions(+)
include/system/iommufd.h +4 -0
@@ -112,6 +112,10 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id, uint64_t addr, uint64_t length, uint32_t *out_hw_queue_id, Error **errp); +bool iommufd_backend_viommu_mmap (IOMMUFDBackend *be,uint32_t viommu_id, +uint64_t size,off_t offset,void **out_ptr, +Error **errp); +bool iommufd_backend_set_dirty_tracking (IOMMUFDBackend *be,uint32_t hwpt_id,bool start,Error **errp); bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +22 -0
@@ -580,6 +580,28 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,return true; } +/* +* Helper to mmap HW MMIO regions exposed via iommufd for a vIOMMU instance. +* The caller is responsible for unmapping the mapped region. +*/ +bool iommufd_backend_viommu_mmap (IOMMUFDBackend *be,uint32_t viommu_id, +uint64_t size,off_t offset,void **out_ptr, +Error **errp) +{ +g_assert (viommu_id); +g_assert (out_ptr); + + *out_ptr =mmap (NULL , size,PROT_READ |PROT_WRITE ,MAP_SHARED , be->fd, + offset); +trace_iommufd_backend_viommu_mmap (be->fd, viommu_id, size, offset); +if (*out_ptr ==MAP_FAILED ) { +error_setg_errno (errp, errno,"IOMMUFD vIOMMU mmap failed" ); +return false; + } + +return true; +} +bool host_iommu_device_iommufd_attach_hwpt (HostIOMMUDeviceIOMMUFD *hiodi,uint32_t hwpt_id,Error **errp) {
backends/trace-events +1 -0
@@ -25,6 +25,7 @@ iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)" iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)" iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)" +iommufd_backend_viommu_mmap(int iommufd, uint32_t viommu_id, uint64_t size, uint64_t offset) " iommufd=%d viommu_id=%u size=0x%"PRIx64" offset=0x%"PRIx64 # igvm-cfg.c igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
PATCH v5 5/32 system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq
In reply to Shameer KolothumThe viommu field is assigned but never used. Callers freeing the veventq already have access to the IOMMUFDViommu object through other references, so this field is redundant.
Removing it also simplifies upcoming changes where veventq is allocated based on the viommu id before the IOMMUFDViommu object is created (e.g. vendor CMDQV-based veventq allocation).
No functional change.
--- include/system/iommufd.h | 1 - hw/arm/smmuv3-accel.c | 1 - 2 files changed, 2 deletions(-)
include/system/iommufd.h +0 -1
@@ -58,7 +58,6 @@ typedef struct IOMMUFDVdev {/* Virtual event queue interface for a vIOMMU */ typedef struct IOMMUFDVeventq { - IOMMUFDViommu *viommu; uint32_t veventq_id; uint32_t veventq_fd; uint32_t last_event_seq; /* Sequence number of last processed event */
hw/arm/smmuv3-accel.c +0 -1
@@ -553,7 +553,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp) veventq =g_new0 (IOMMUFDVeventq,1 ); veventq->veventq_id = veventq_id; veventq->veventq_fd = veventq_fd; - veventq->viommu = accel->viommu; accel->veventq = veventq;/* Set up event handler for veventq fd */
Signature · 1 line
2.43.0
PATCH v5 6/32 hw/arm/smmuv3-accel: Introduce CMDQV ops interface
In reply to Shameer KolothumCommand Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.
For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).
The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.
CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.
This provides an extensible design to support future vendor-specific CMDQV implementations.
No functional change.
--- hw/arm/smmuv3-accel.h | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
hw/arm/smmuv3-accel.h +35 -0
@@ -10,11 +10,45 @@#define HW_ARM_SMMUV3_ACCEL_H #include "hw/arm/smmu-common.h" +#include "hw/arm/smmuv3.h" #include "system/iommufd.h" #ifdef CONFIG_LINUX #include <linux/iommufd.h> #endif +/* +* CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to +* support multiple VCMDQs with virtualization capabilities. +* CMDQV specific behavior is factored behind this ops interface. +*/ +typedef struct SMMUv3AccelCmdqvOps { +/** +* @probe: Optional callback. Vendor-specific device probing. +*/ +bool (*probe)(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp); +/** +* @init: Optional callback. Initialize CMDQV hardware. +*/ +bool (*init)(SMMUv3State *s,Error **errp); +/** +* @alloc_viommu: Mandatory when probe is implemented. +* Allocate CMDQV viommu resources. +*/ +bool (*alloc_viommu)(SMMUv3State *s, +HostIOMMUDeviceIOMMUFD *idev, +uint32_t *out_viommu_id, +Error **errp); +/** +* @free_viommu: Optional callback. Free CMDQV viommu resources. +* If NULL, the viommu_id is freed directly via iommufd_backend_free_id(). +*/ +void (*free_viommu)(SMMUv3State *s); +/** +* @reset: Optional callback. Reset CMDQV state. +*/ +void (*reset)(SMMUv3State *s); +}SMMUv3AccelCmdqvOps ; +/* * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object. * Holds bypass and abort proxy HWPT IDs used for device attachment.
@@ -27,6 +61,7 @@ typedef struct SMMUv3AccelState { QLIST_HEAD(, SMMUv3AccelDevice) device_list;bool auto_mode;bool auto_finalised; +const SMMUv3AccelCmdqvOps *cmdqv_ops; } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
Signature · 1 line
2.43.0
Re: [PATCH v5 06/32] hw/arm/smmuv3-accel: Introduce CMDQV ops interface
Quoted 47 lines · Shameer Kolothum
Command Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.
For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).
The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.
CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.
This provides an extensible design to support future vendor-specific CMDQV implementations.
No functional change.
Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/smmuv3-accel.h | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)hw/arm/smmuv3-accel.h +7
-0@@ -10,11 +10,45 @@#define HW_ARM_SMMUV3_ACCEL_H #include "hw/arm/smmu-common.h" +#include "hw/arm/smmuv3.h" #include "system/iommufd.h" #ifdef CONFIG_LINUX #include <linux/iommufd.h> #endif +/* +* CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to +* support multiple VCMDQs with virtualization capabilities. +* CMDQV specific behavior is factored behind this ops interface. +*/ +typedef struct SMMUv3AccelCmdqvOps {
hw/arm/smmuv3-accel.h line 34
Quoted 10 lines
hw/arm/smmuv3-accel.h +10
-0+/** +* @probe: Optional callback. Vendor-specific device probing. +*/ +bool (*probe)(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp); +/** +* @init: Optional callback. Initialize CMDQV hardware. +*/ +bool (*init)(SMMUv3State *s,Error **errp); +/** +* @alloc_viommu: Mandatory when probe is implemented.
see my comment on 8/32. Aren't probe and alloc_viommu somehow mandatory?
Eric
Quoted 28 lines
hw/arm/smmuv3-accel.h +18
-0+* Allocate CMDQV viommu resources. +*/ +bool (*alloc_viommu)(SMMUv3State *s, +HostIOMMUDeviceIOMMUFD *idev, +uint32_t *out_viommu_id, +Error **errp); +/** +* @free_viommu: Optional callback. Free CMDQV viommu resources. +* If NULL, the viommu_id is freed directly via iommufd_backend_free_id(). +*/ +void (*free_viommu)(SMMUv3State *s); +/** +* @reset: Optional callback. Reset CMDQV state. +*/ +void (*reset)(SMMUv3State *s); +} SMMUv3AccelCmdqvOps; +/* * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object. * Holds bypass and abort proxy HWPT IDs used for device attachment. @@ -27,6 +61,7 @@ typedef struct SMMUv3AccelState { QLIST_HEAD(, SMMUv3AccelDevice) device_list;bool auto_mode;bool auto_finalised; +const SMMUv3AccelCmdqvOps *cmdqv_ops; } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
PATCH v5 7/32 hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub
In reply to Shameer KolothumIntroduce a Tegra241 CMDQV backend that plugs into the SMMUv3 accelerated CMDQV ops interface.
This patch wires up the Tegra241 CMDQV backend and provides a stub implementation for CMDQV probe, initialization, vIOMMU allocation and reset handling.
Functional CMDQV support is added in follow-up patches.
--- hw/arm/tegra241-cmdqv.h | 15 ++++++++++ hw/arm/tegra241-cmdqv-stubs.c | 16 ++++++++++ hw/arm/tegra241-cmdqv.c | 56 +++++++++++++++++++++++++++++++++++ hw/arm/Kconfig | 5 ++++ hw/arm/meson.build | 2 ++ 5 files changed, 94 insertions(+) create mode 100644 hw/arm/tegra241-cmdqv.h create mode 100644 hw/arm/tegra241-cmdqv-stubs.c create mode 100644 hw/arm/tegra241-cmdqv.c
hw/arm/tegra241-cmdqv.h new file +15 -0
@@ -0,0 +1,15 @@ +/* +* Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved +* NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3 +* +* Written by Nicolin Chen, Shameer Kolothum +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_ARM_TEGRA241_CMDQV_H +#define HW_ARM_TEGRA241_CMDQV_H + +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void ); + +#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv-stubs.c new file +16 -0
@@ -0,0 +1,16 @@ +/* +* Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved +* +* Stubs for Tegra241 CMDQ-Virtualization extension for SMMUv3 +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "smmuv3-accel.h" +#include "hw/arm/tegra241-cmdqv.h" + +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void ) +{ +return NULL ; +}
hw/arm/tegra241-cmdqv.c new file +56 -0
56 lines not on this page · raw message
hw/arm/Kconfig +5 -0
@@ -642,6 +642,10 @@ config FSL_IMX8MM_EVK depends on TCG select FSL_IMX8MM +config TEGRA241_CMDQV + bool + depends on ARM_SMMUV3_ACCEL + config ARM_SMMUV3_ACCEL bool depends on ARM_SMMUV3
@@ -649,6 +653,7 @@ config ARM_SMMUV3_ACCEL config ARM_SMMUV3 bool select ARM_SMMUV3_ACCEL if IOMMUFD + imply TEGRA241_CMDQV config FSL_IMX6UL bool
hw/arm/meson.build +2 -0
@@ -84,6 +84,8 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MM_EVK', if_true: files('imx8mm-evk.c')) arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c')) arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c')) stub_ss.add(files('smmuv3-accel-stubs.c')) +arm_common_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c')) +stub_ss.add(files('tegra241-cmdqv-stubs.c')) arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c')) arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c')) arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
Signature · 1 line
2.43.0
PATCH v5 8/32 hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle
In reply to Shameer KolothumAdd support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.
If set to OFF, CMDQV is not used and the default IOMMUFD-backed allocation path is taken.
If set to AUTO, QEMU attempts to probe a CMDQV backend during device setup. If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.
If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.
When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.
The current implementation wires up the Tegra241 CMDQV backend through the generic ops interface. Functional CMDQV behaviour is added in subsequent patches.
No functional change.
--- include/hw/arm/smmuv3.h | 2 + hw/arm/smmuv3-accel.c | 96 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 91 insertions(+), 7 deletions(-)
include/hw/arm/smmuv3.h +2 -0
@@ -74,6 +74,8 @@ struct SMMUv3State {OnOffAuto ats;OasMode oas;SsidSizeMode ssidsize; +/* SMMU CMDQV extension */ +OnOffAuto cmdqv;Notifier machine_done; };
hw/arm/smmuv3-accel.c +89 -7
137 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 08/32] hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle
Hi SHameer,
Quoted 66 lines · Shameer Kolothum
Add support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.
If set to OFF, CMDQV is not used and the default IOMMUFD-backed allocation path is taken.
If set to AUTO, QEMU attempts to probe a CMDQV backend during device setup. If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.
If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.
When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.
The current implementation wires up the Tegra241 CMDQV backend through the generic ops interface. Functional CMDQV behaviour is added in subsequent patches.
No functional change.
Reviewed-by Eric Auger <[email protected]>Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- include/hw/arm/smmuv3.h | 2 + hw/arm/smmuv3-accel.c | 96 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 91 insertions(+), 7 deletions(-)include/hw/arm/smmuv3.h +2
-0@@ -74,6 +74,8 @@ struct SMMUv3State {OnOffAuto ats;OasMode oas;SsidSizeMode ssidsize; +/* SMMU CMDQV extension */ +OnOffAuto cmdqv;Notifier machine_done; };hw/arm/smmuv3-accel.c +2
-0@@ -19,6 +19,7 @@#include "smmuv3-internal.h" #include "smmuv3-accel.h" #include "system/system.h" +#include "tegra241-cmdqv.h" /* * The root region aliases the global system memory, and shared_as_sysmem @@ -570,6 +571,7 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, Error **errp) {SMMUv3AccelState *accel = s->s_accel; +const SMMUv3AccelCmdqvOps *cmdqv_ops = accel->cmdqv_ops;struct iommu_hwpt_arm_smmuv3 bypass_data = { .ste = {SMMU_STE_CFG_BYPASS |SMMU_STE_VALID ,0x0ULL }, };
hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, · line 586
Quoted 9 lines
hw/arm/smmuv3-accel.c +2
-4@@ -580,10 +582,17 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,uint32_t viommu_id, hwpt_id;IOMMUFDViommu *viommu; -if (!iommufd_backend_alloc_viommu (hiodi->iommufd, hiodi->devid, -IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , s2_hwpt_id, -NULL ,0 , &viommu_id, errp)) { -return false; +if (cmdqv_ops) { +if (!cmdqv_ops->alloc_viommu (s, hiodi, &viommu_id, errp)) {
the doc comment in 6/32 says that alloc_iommu is mandatory if probe is implemented. So it looks there are situations where alloc_iommu may be null.
But actually given the implementation of smmuv3_accel_probe_cmdqv below, the ops is only non NULL if probe has succeeded. So shouldn't probe be also mandatory?
Quoted 64 lines
hw/arm/smmuv3-accel.c +51
-1+return false; + } + }else { +if (!iommufd_backend_alloc_viommu (hiodi->iommufd, hiodi->devid, +IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , + s2_hwpt_id,NULL ,0 , &viommu_id, + errp)) { +return false; + } } viommu =g_new0 (IOMMUFDViommu,1 );@@ -629,12 +638,72 @@ free_bypass_hwpt: free_abort_hwpt:iommufd_backend_free_id (hiodi->iommufd, accel->abort_hwpt_id); free_viommu: -iommufd_backend_free_id (hiodi->iommufd, viommu->viommu_id); +if (cmdqv_ops && cmdqv_ops->free_viommu) { + cmdqv_ops->free_viommu (s); + }else { +iommufd_backend_free_id (hiodi->iommufd, viommu->viommu_id); + }g_free (viommu); accel->viommu =NULL ;return false; } +static const SMMUv3AccelCmdqvOps * +smmuv3_accel_probe_cmdqv (SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev, +Error **errp) +{ +const SMMUv3AccelCmdqvOps *ops =tegra241_cmdqv_get_ops (); + +if (!ops || !ops->probe) { +error_setg (errp,"No CMDQV ops found" ); +return NULL ; + } + +if (!ops->probe (s, idev, errp)) { +return NULL ; + } +return ops; +} + +static bool +smmuv3_accel_select_cmdqv (SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev, +Error **errp) +{ +const SMMUv3AccelCmdqvOps *ops =NULL ; + +if (s->s_accel->cmdqv_ops) { +return true; + } + +switch (s->cmdqv) { +case ON_OFF_AUTO_OFF : + s->s_accel->cmdqv_ops =NULL ; +return true; +case ON_OFF_AUTO_AUTO : + ops =smmuv3_accel_probe_cmdqv (s, idev,NULL ); +break ; +case ON_OFF_AUTO_ON : + ops =smmuv3_accel_probe_cmdqv (s, idev, errp); +if (!ops) {
hw/arm/smmuv3-accel.c free_bypass_hwpt: · line 697
Quoted 10 lines
hw/arm/smmuv3-accel.c +10
-0+error_append_hint (errp,"CMDQV requested but not supported" ); +return false; + } +break ; +default : +g_assert_not_reached (); + } + +if (ops) { +g_assert (ops->alloc_viommu);
I think I would move the assert at the end of
smmuv3_accel_probe_cmdqv()
Quoted 33 lines
hw/arm/smmuv3-accel.c +17
-2+ } + +if (ops && ops->init && !ops->init (s, errp)) { +return false; + } + s->s_accel->cmdqv_ops = ops; +return true; +} +static bool smmuv3_accel_set_iommu_device (PCIBus *bus,void *opaque,int devfn,HostIOMMUDevice *hiod,Error **errp) {@@ -669,6 +738,10 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn, goto done; } +if (!smmuv3_accel_select_cmdqv (s, hiodi, errp)) { +return false; + } +if (!smmuv3_accel_alloc_viommu (s, hiodi, errp)) {error_append_hint (errp,"Unable to alloc vIOMMU: hiodi devid 0x%x: " , hiodi->devid);@@ -946,8 +1019,17 @@ bool smmuv3_accel_attach_gbpa_hwpt(SMMUv3State *s, Error **errp)void smmuv3_accel_reset (SMMUv3State *s) { -/* Attach a HWPT based on GBPA reset value */ -smmuv3_accel_attach_gbpa_hwpt (s,NULL ); +SMMUv3AccelState *accel = s->s_accel; + +if (!accel) { +return ;
hw/arm/smmuv3-accel.c bool smmuv3_accel_attach_gbpa_hwpt(SMMUv3State *s, Error **errp) · line 1035
Quoted 10 lines
hw/arm/smmuv3-accel.c +7
-0+ } +/* Attach a HWPT based on GBPA reset value */ +smmuv3_accel_attach_gbpa_hwpt (s,NULL ); + +if (accel->cmdqv_ops && accel->cmdqv_ops->reset) { + accel->cmdqv_ops->reset (s); + } }static void smmuv3_accel_as_init (SMMUv3State *s)
Eric
RE: [PATCH v5 08/32] hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle
Hi Eric,
Quoted 102 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 06:27
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 08/32] hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycleExternal email: Use caution opening links or attachments
Hi SHameer,
Quoted 4 lines · Shameer Kolothum
Add support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.
If set to OFF, CMDQV is not used and the default IOMMUFD-backed
allocation
Quoted 3 lines
path is taken.
If set to AUTO, QEMU attempts to probe a CMDQV backend during device
setup.
Quoted 4 lines
If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.
Quoted 8 lines
If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.
When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.
The current implementation wires up the Tegra241 CMDQV backend
through the
Quoted 11 lines
generic ops interface. Functional CMDQV behaviour is added in subsequent patches.
No functional change.
Reviewed-by Eric Auger <[email protected]>Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- include/hw/arm/smmuv3.h | 2 + hw/arm/smmuv3-accel.c | 96++++++++++++++++++++++++++++++++++++++---
Quoted 20 lines
2 files changed, 91 insertions(+), 7 deletions(-)include/hw/arm/smmuv3.h +2
-0@@ -74,6 +74,8 @@ struct SMMUv3State {OnOffAuto ats;OasMode oas;SsidSizeMode ssidsize; +/* SMMU CMDQV extension */ +OnOffAuto cmdqv;Notifier machine_done; };hw/arm/smmuv3-accel.c +0
-0hw/arm/smmuv3-accel.c line 25
Quoted 7 lines
hw/arm/smmuv3-accel.c +1
-0@@ -19,6 +19,7 @@#include "smmuv3-internal.h" #include "smmuv3-accel.h" #include "system/system.h" +#include "tegra241-cmdqv.h" /* * The root region aliases the global system memory, and
shared_as_sysmemhw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s,
Quoted 1 line
hw/arm/smmuv3-accel.c +0
-0@@ -570,6 +571,7 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *hiodi,
hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s,
Quoted 8 lines
hw/arm/smmuv3-accel.c +1
-0Error **errp) {SMMUv3AccelState *accel = s->s_accel; +const SMMUv3AccelCmdqvOps *cmdqv_ops = accel->cmdqv_ops;struct iommu_hwpt_arm_smmuv3 bypass_data = { .ste = {SMMU_STE_CFG_BYPASS |SMMU_STE_VALID ,0x0ULL }, };@@ -580,10 +582,17 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *hiodi,
hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
Quoted 5 lines
hw/arm/smmuv3-accel.c +0
-2uint32_t viommu_id, hwpt_id;IOMMUFDViommu *viommu; - if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid, - IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
s2_hwpt_id,hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, · line 586
Quoted 4 lines
hw/arm/smmuv3-accel.c +2
-2- NULL, 0, &viommu_id, errp)) { -return false; +if (cmdqv_ops) { +if (!cmdqv_ops->alloc_viommu (s, hiodi, &viommu_id, errp)) {
Quoted 7 lines
the doc comment in 6/32 says that
alloc_iommuis mandatory if probe is implemented. So it looks there are situations wherealloc_iommumay be null.But actually given the implementation of
smmuv3_accel_probe_cmdqvbelow,
the ops is only non NULL if probe has succeeded. So shouldn't probe be also mandatory?
Yeah. You are right. Probe is mandatory if ops is non NULL. I will revise the comments.
Quoted 78 lines
Quoted 19 lines
hw/arm/smmuv3-accel.c +11
-1+return false; + } + }else { +if (!iommufd_backend_alloc_viommu (hiodi->iommufd, hiodi->devid, +IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , + s2_hwpt_id,NULL ,0 , &viommu_id, + errp)) { +return false; + } } viommu =g_new0 (IOMMUFDViommu,1 );@@ -629,12 +638,72 @@ free_bypass_hwpt: free_abort_hwpt:iommufd_backend_free_id (hiodi->iommufd, accel->abort_hwpt_id); free_viommu: -iommufd_backend_free_id (hiodi->iommufd, viommu->viommu_id); +if (cmdqv_ops && cmdqv_ops->free_viommu) { + cmdqv_ops->free_viommu (s);hw/arm/smmuv3-accel.c free_bypass_hwpt: · line 652
Quoted 10 lines
hw/arm/smmuv3-accel.c +5
-0+ }else { +iommufd_backend_free_id (hiodi->iommufd, viommu->viommu_id); + }g_free (viommu); accel->viommu =NULL ;return false; } +static const SMMUv3AccelCmdqvOps * +smmuv3_accel_probe_cmdqv(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,
Quoted 7 lines
hw/arm/smmuv3-accel.c +7
-0+ Error **errp) +{ +const SMMUv3AccelCmdqvOps *ops =tegra241_cmdqv_get_ops (); + +if (!ops || !ops->probe) { +error_setg (errp,"No CMDQV ops found" ); +return NULL ;hw/arm/smmuv3-accel.c free_bypass_hwpt:
Quoted 10 lines
hw/arm/smmuv3-accel.c +10
-0+ } + +if (!ops->probe (s, idev, errp)) { +return NULL ; + } +return ops; +} + +static bool +smmuv3_accel_select_cmdqv(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,
Quoted 18 lines
hw/arm/smmuv3-accel.c +18
-0+ Error **errp) +{ +const SMMUv3AccelCmdqvOps *ops =NULL ; + +if (s->s_accel->cmdqv_ops) { +return true; + } + +switch (s->cmdqv) { +case ON_OFF_AUTO_OFF : + s->s_accel->cmdqv_ops =NULL ; +return true; +case ON_OFF_AUTO_AUTO : + ops =smmuv3_accel_probe_cmdqv (s, idev,NULL ); +break ; +case ON_OFF_AUTO_ON : + ops =smmuv3_accel_probe_cmdqv (s, idev, errp); +if (!ops) {hw/arm/smmuv3-accel.c free_bypass_hwpt: · line 697
Quoted 10 lines
hw/arm/smmuv3-accel.c +10
-0+error_append_hint (errp,"CMDQV requested but not supported" ); +return false; + } +break ; +default : +g_assert_not_reached (); + } + +if (ops) { +g_assert (ops->alloc_viommu);I think I would move the assert at the end of
Quoted 1 line
smmuv3_accel_probe_cmdqv()
Agree. I will change it into:
smmuv3_accel_probe_cmdqv()
{
....
if (!ops) {
error_setg(errp, "No CMDQV ops found");
return NULL;
}
g_assert(ops->probe);
g_assert(ops->alloc_viommu);
if (!ops->probe(s, idev, errp)) {
return NULL;
}
return ops;}
Thanks,
Shameer
PATCH v5 9/32 hw/arm/virt: Use stored SMMUv3 device list for IORT build
In reply to Shameer KolothumIntroduce a GPtrArray in VirtMachineState to track all SMMUv3 devices created on the virt machine, and use it when building the IORT table instead of relying on object_child_foreach_recursive() walks of the object tree.
This avoids recursive object traversal and provides a foundation for subsequent patches that need direct access to SMMUv3 instances for CMDQV-related handling.
No functional change. No bios-tables qtest failures observed.
--- include/hw/arm/virt.h | 1 + hw/arm/virt-acpi-build.c | 70 ++++++++++++++++++---------------------- hw/arm/virt.c | 3 ++ 3 files changed, 35 insertions(+), 39 deletions(-)
include/hw/arm/virt.h +1 -0
@@ -207,6 +207,7 @@ struct VirtMachineState {MemoryRegion *sysmem;MemoryRegion *secure_sysmem;bool pci_preserve_config; +GPtrArray *smmuv3_devices; };#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
hw/arm/virt-acpi-build.c +31 -39
87 lines not on this page · raw message
hw/arm/virt.c +3 -0
@@ -3865,6 +3865,7 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, }create_smmuv3_dev_dtb (vms, dev, bus, errp); +g_ptr_array_add (vms->smmuv3_devices, dev); } }
@@ -4319,6 +4320,8 @@ static void virt_instance_init(Object *obj) vms->oem_id =g_strndup (ACPI_BUILD_APPNAME6 ,6 ); vms->oem_table_id =g_strndup (ACPI_BUILD_APPNAME8 ,8 );cxl_machine_init (obj, &vms->cxl_devices_state); + + vms->smmuv3_devices =g_ptr_array_new_with_free_func (NULL ); }static void virt_instance_finalize (Object *obj)
Signature · 1 line
2.43.0
PATCH v5 10/32 hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support
In reply to Shameer KolothumUse IOMMU_GET_HW_INFO to query host support for Tegra241 CMDQV.
Validate the returned data type, version, and minimum number of vCMDQs and SIDs per Tegra241 CMDQ Virtual Interface(VI). Fail the probe if the host does not meet these requirements.
The QEMU model supports one Virtual Interface(VI) per VM with 2 vCMDQs and 16 SIDs per VI, so the probe ensures the host implementation is compatible with these limits.
--- hw/arm/tegra241-cmdqv.h | 4 ++++ hw/arm/tegra241-cmdqv.c | 32 ++++++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.h +4 -0
@@ -10,6 +10,10 @@#ifndef HW_ARM_TEGRA241_CMDQV_H #define HW_ARM_TEGRA241_CMDQV_H +#define CMDQV_VER 1 +#define CMDQV_NUM_CMDQ_LOG2 1 +#define CMDQV_NUM_SID_PER_VI_LOG2 4 +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +30 -2
@@ -38,8 +38,36 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)static bool tegra241_cmdqv_probe (SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp) { -error_setg (errp,"NVIDIA Tegra241 CMDQV is unsupported" ); -return false; +uint32_t data_type =IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV ; +struct iommu_hw_info_tegra241_cmdqv cmdqv_info; +uint64_t caps; + +if (!iommufd_backend_get_device_info (idev->iommufd, idev->devid, &data_type, + &cmdqv_info,sizeof (cmdqv_info), &caps, +NULL , errp)) { +return false; + } +if (data_type !=IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV ) { +error_setg (errp,"Host CMDQV: unexpected data type %u (expected %u)" , + data_type,IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV ); +return false; + } +if (cmdqv_info.version !=CMDQV_VER ) { +error_setg (errp,"Host CMDQV: unsupported version %u (expected %u)" , + cmdqv_info.version,CMDQV_VER ); +return false; + } +if (cmdqv_info.log2vcmdqs <CMDQV_NUM_CMDQ_LOG2 ) { +error_setg (errp,"Host CMDQV: insufficient vCMDQs log2=%u (need >= %u)" , + cmdqv_info.log2vcmdqs,CMDQV_NUM_CMDQ_LOG2 ); +return false; + } +if (cmdqv_info.log2vsids <CMDQV_NUM_SID_PER_VI_LOG2 ) { +error_setg (errp,"Host CMDQV: insufficient SIDs log2=%u (need >= %u)" , + cmdqv_info.log2vsids,CMDQV_NUM_SID_PER_VI_LOG2 ); +return false; + } +return true; }static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = {
Signature · 1 line
2.43.0
PATCH v5 11/32 hw/arm/tegra241-cmdqv: Implement CMDQV init
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Tegra241 CMDQV extends SMMUv3 with support for virtual command queues (VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:
0x00000 (CMDQ-V Config page) 0x10000 (CMDQ-V CMDQ Page0) 0x20000 (CMDQ-V CMDQ Page1) 0x30000 (Virtual Interface Page0) 0x40000 (Virtual Interface Page1)
This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.
The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.
The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.
--- hw/arm/smmuv3-accel.h | 1 + hw/arm/tegra241-cmdqv.h | 18 ++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 38 ++++++++++++++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 2 deletions(-)
hw/arm/smmuv3-accel.h +1 -0
@@ -62,6 +62,7 @@ typedef struct SMMUv3AccelState {bool auto_mode;bool auto_finalised;const SMMUv3AccelCmdqvOps *cmdqv_ops; +void *cmdqv;/* vendor specific CMDQV state */ } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
hw/arm/tegra241-cmdqv.h +18 -0
@@ -14,6 +14,24 @@#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4 +/* +* Tegra241 CMDQV MMIO layout (64KB pages) +* +* 0x00000 (CMDQ-V Config page) +* 0x10000 (CMDQ-V CMDQ Page0) +* 0x20000 (CMDQ-V CMDQ Page1) +* 0x30000 (Virtual Interface Page0) +* 0x40000 (Virtual Interface Page1) +*/ +#define TEGRA241_CMDQV_IO_LEN 0x50000 + +typedef struct Tegra241CMDQV { +struct iommu_viommu_tegra241_cmdqv cmdqv_data; +SMMUv3AccelState *s_accel; +MemoryRegion mmio_cmdqv; +qemu_irq irq; +}Tegra241CMDQV ; +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +36 -2
@@ -13,6 +13,16 @@#include "smmuv3-accel.h" #include "tegra241-cmdqv.h" +static uint64_t tegra241_cmdqv_read_mmio (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static void tegra241_cmdqv_write_mmio (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) { }
@@ -29,10 +39,34 @@ static void tegra241_cmdqv_reset(SMMUv3State *s) { } +static const MemoryRegionOps mmio_cmdqv_ops = { + .read = tegra241_cmdqv_read_mmio, + .write = tegra241_cmdqv_write_mmio, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; +static bool tegra241_cmdqv_init (SMMUv3State *s,Error **errp) { -error_setg (errp,"NVIDIA Tegra241 CMDQV is unsupported" ); -return false; +SysBusDevice *sbd =SYS_BUS_DEVICE (OBJECT (s)); +SMMUv3AccelState *accel = s->s_accel; +Tegra241CMDQV *cmdqv; + + cmdqv =g_new0 (Tegra241CMDQV,1 ); +memory_region_init_io (&cmdqv->mmio_cmdqv,OBJECT (s), &mmio_cmdqv_ops, cmdqv, +"tegra241-cmdqv" ,TEGRA241_CMDQV_IO_LEN ); +sysbus_init_mmio (sbd, &cmdqv->mmio_cmdqv); +sysbus_init_irq (sbd, &cmdqv->irq); + cmdqv->s_accel = accel; + accel->cmdqv = cmdqv; +return true; } static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
Signature · 1 line
2.43.0
Re: [PATCH v5 11/32] hw/arm/tegra241-cmdqv: Implement CMDQV init
Hello Shameer,
Quoted 58 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Tegra241 CMDQV extends SMMUv3 with support for virtual command queues (VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:
0x00000 (CMDQ-V Config page) 0x10000 (CMDQ-V CMDQ Page0) 0x20000 (CMDQ-V CMDQ Page1) 0x30000 (Virtual Interface Page0) 0x40000 (Virtual Interface Page1)This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.
The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.
The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.
Signed-off-by Nicolin Chen <[email protected]>Reviewed-by Eric Auger <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/smmuv3-accel.h | 1 + hw/arm/tegra241-cmdqv.h | 18 ++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 38 ++++++++++++++++++++++++++++++++++++-- 3 files changed, 55 insertions(+), 2 deletions(-)hw/arm/smmuv3-accel.h +1
-0@@ -62,6 +62,7 @@ typedef struct SMMUv3AccelState {bool auto_mode;bool auto_finalised;const SMMUv3AccelCmdqvOps *cmdqv_ops; +void *cmdqv;/* vendor specific CMDQV state */ } SMMUv3AccelState; typedef struct SMMUS1Hwpt {hw/arm/tegra241-cmdqv.h +3
-0@@ -14,6 +14,24 @@#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4 +/* +* Tegra241 CMDQV MMIO layout (64KB pages) +*
hw/arm/tegra241-cmdqv.h line 29
Quoted 10 lines
hw/arm/tegra241-cmdqv.h +10
-0+* 0x00000 (CMDQ-V Config page) +* 0x10000 (CMDQ-V CMDQ Page0) +* 0x20000 (CMDQ-V CMDQ Page1) +* 0x30000 (Virtual Interface Page0) +* 0x40000 (Virtual Interface Page1) +*/ +#define TEGRA241_CMDQV_IO_LEN 0x50000 + +typedef struct Tegra241CMDQV { + struct iommu_viommu_tegra241_cmdqv cmdqv_data;
This won't build on Windows.
C.
Quoted 66 lines
hw/arm/tegra241-cmdqv.h +5
-0+SMMUv3AccelState *s_accel; +MemoryRegion mmio_cmdqv; + qemu_irq irq; +} Tegra241CMDQV; +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +36
-2@@ -13,6 +13,16 @@#include "smmuv3-accel.h" #include "tegra241-cmdqv.h" +static uint64_t tegra241_cmdqv_read_mmio (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static void tegra241_cmdqv_write_mmio (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) { }@@ -29,10 +39,34 @@ static void tegra241_cmdqv_reset(SMMUv3State *s) { } +static const MemoryRegionOps mmio_cmdqv_ops = { + .read = tegra241_cmdqv_read_mmio, + .write = tegra241_cmdqv_write_mmio, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; +static bool tegra241_cmdqv_init (SMMUv3State *s,Error **errp) { -error_setg (errp,"NVIDIA Tegra241 CMDQV is unsupported" ); -return false; +SysBusDevice *sbd =SYS_BUS_DEVICE (OBJECT (s)); +SMMUv3AccelState *accel = s->s_accel; +Tegra241CMDQV *cmdqv; + + cmdqv =g_new0 (Tegra241CMDQV,1 ); +memory_region_init_io (&cmdqv->mmio_cmdqv,OBJECT (s), &mmio_cmdqv_ops, cmdqv, +"tegra241-cmdqv" ,TEGRA241_CMDQV_IO_LEN ); +sysbus_init_mmio (sbd, &cmdqv->mmio_cmdqv); +sysbus_init_irq (sbd, &cmdqv->irq); + cmdqv->s_accel = accel; + accel->cmdqv = cmdqv; +return true; } static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
RE: [PATCH v5 11/32] hw/arm/tegra241-cmdqv: Implement CMDQV init
Hi Cédric,
Quoted 76 lines
-----Original Message-----
From: Cédric Le Goater <[email protected]>
Sent: 21 May 2026 09:19
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 11/32] hw/arm/tegra241-cmdqv: Implement CMDQV initExternal email: Use caution opening links or attachments
Hello Shameer,
Quoted 3 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Tegra241 CMDQV extends SMMUv3 with support for virtual command
queues
Quoted 22 lines
(VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:
0x00000 (CMDQ-V Config page) 0x10000 (CMDQ-V CMDQ Page0) 0x20000 (CMDQ-V CMDQ Page1) 0x30000 (Virtual Interface Page0) 0x40000 (Virtual Interface Page1)This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.
The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.
The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.
Quoted 8 lines
Signed-off-by Nicolin Chen <[email protected]>Reviewed-by Eric Auger <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/smmuv3-accel.h | 1 + hw/arm/tegra241-cmdqv.h | 18 ++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 38++++++++++++++++++++++++++++++++++++--
Quoted 22 lines
3 files changed, 55 insertions(+), 2 deletions(-)hw/arm/smmuv3-accel.h index renamed from hw/arm/smmuv3-accel.h +0
-0
f82a7112d8..49c10535cf100644hw/arm/smmuv3-accel.h +1
-0@@ -62,6 +62,7 @@ typedef struct SMMUv3AccelState {bool auto_mode;bool auto_finalised;const SMMUv3AccelCmdqvOps *cmdqv_ops; +void *cmdqv;/* vendor specific CMDQV state */ } SMMUv3AccelState; typedef struct SMMUS1Hwpt {hw/arm/tegra241-cmdqv.h index renamed from hw/arm/tegra241-cmdqv.h +0
-0
38c8b27b4d..030f5758d5100644hw/arm/tegra241-cmdqv.h +0
-0@@ -14,6 +14,24 @@#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4
Quoted 14 lines
hw/arm/tegra241-cmdqv.h line 29
Quoted 12 lines
hw/arm/tegra241-cmdqv.h +12
-0+/* +* Tegra241 CMDQV MMIO layout (64KB pages) +* +* 0x00000 (CMDQ-V Config page) +* 0x10000 (CMDQ-V CMDQ Page0) +* 0x20000 (CMDQ-V CMDQ Page1) +* 0x30000 (Virtual Interface Page0) +* 0x40000 (Virtual Interface Page1) */ #define +TEGRA241_CMDQV_IO_LEN 0x50000 + +typedef struct Tegra241CMDQV { + struct iommu_viommu_tegra241_cmdqv cmdqv_data;This won't build on Windows.
Oops..I did that again!. Will fix it in v6.
Thanks,
Shameer
PATCH v5 12/32 hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus
In reply to Shameer KolothumSMMUv3 devices with acceleration may enable CMDQV extensions after device realize. In that case, additional MMIO regions and IRQ lines may be registered but not yet mapped to the platform bus.
Ensure SMMUv3 device resources are linked to the platform bus during machine_done().
This is safe to do unconditionally since the platform bus helpers skip resources that are already mapped.
--- hw/arm/virt.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
hw/arm/virt.c +22 -0
@@ -2361,6 +2361,25 @@ static void virt_build_smbios(VirtMachineState *vms) } } +/* +* SMMUv3 devices with acceleration may enable CMDQV extensions +* after device realize. In that case, additional MMIO regions and +* IRQ lines may be registered but not yet mapped to the platform bus. +* +* Ensure all resources are linked to the platform bus before final +* machine setup. +*/ + +static void virt_smmuv3_dev_link_cmdqv (VirtMachineState *vms) +{ +for (int i =0 ; i < vms->smmuv3_devices->len; i++) { +DeviceState *dev =g_ptr_array_index (vms->smmuv3_devices, i); + +platform_bus_link_device (PLATFORM_BUS_DEVICE (vms->platform_bus_dev), +SYS_BUS_DEVICE (dev)); + } +} +static void virt_machine_done (Notifier *notifier,void *data) {
@@ -2377,6 +2396,9 @@ void virt_machine_done(Notifier *notifier, void *data)if (vms->cxl_devices_state.is_enabled) {cxl_fmws_link_targets (&error_fatal); } + +virt_smmuv3_dev_link_cmdqv (vms); +/* * If the user provided a dtb, we assume the dynamic sysbus nodes * already are integrated there. This corresponds to a use case where
Signature · 1 line
2.43.0
PATCH v5 13/32 hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Replace the stub implementation with real vIOMMU allocation for Tegra241 CMDQV.
Allocate a matching vEVENTQ together with the vIOMMU, since it is specific to the Tegra241 CMDQV vIOMMU and used to receive CMDQV events.
Free both objects on teardown.
--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 48 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.h +1 -0
@@ -30,6 +30,7 @@ typedef struct Tegra241CMDQV {SMMUv3AccelState *s_accel;MemoryRegion mmio_cmdqv;qemu_irq irq; +IOMMUFDVeventq *veventq; } Tegra241CMDQV;const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );
hw/arm/tegra241-cmdqv.c +47 -1
66 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v5 14/32 hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Tegra241 CMDQV exposes control and status registers in the CMDQ-V Config page (offset [0x0, 0x10000)) used to configure virtual command queue allocation and interrupt behavior.
Add read/write emulation for the CMDQ-V Config region ([CMDQV_BASE, CMDQV_CMDQ_BASE]), backed by a simple register cache. This includes CONFIG, PARAM, STATUS, VI error and interrupt maps, CMDQ allocation map and the VINTF0 related registers defined in the CMDQ-V Config space. Only VINTF0 is supported; VINTF1-63 are not.
Dispatch writes on access size: Introduced writel_mmio for 4-byte and writell_mmio for 8-byte. Reads need no split as the MMIO framework masks the returned value to the access size.
--- hw/arm/tegra241-cmdqv.h | 110 +++++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 188 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 4 + 3 files changed, 298 insertions(+), 4 deletions(-)
hw/arm/tegra241-cmdqv.h +110 -0
@@ -10,10 +10,15 @@#ifndef HW_ARM_TEGRA241_CMDQV_H #define HW_ARM_TEGRA241_CMDQV_H +#include "hw/core/registerfields.h" +#define CMDQV_VER 1#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4 +#define TEGRA241_CMDQV_MAX_CMDQ (1U << CMDQV_NUM_CMDQ_LOG2) +#define TEGRA241_CMDQV_MAX_NUM_SID (1U << CMDQV_NUM_SID_PER_VI_LOG2) +/* * Tegra241 CMDQV MMIO layout (64KB pages) *
@@ -31,8 +36,113 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; + +/* CMDQ-V Config page register cache */ +uint32_t config; +uint32_t param; +uint32_t status; +uint32_t vi_err_map[2 ]; +uint32_t vi_int_mask[2 ]; +uint32_t cmdq_err_map[4 ]; +uint32_t cmdq_alloc_map[TEGRA241_CMDQV_MAX_CMDQ ]; + +/* VINTF0 register cache (within CMDQ-V Config page) */ +uint32_t vintf_config; +uint32_t vintf_status; +uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID ]; +uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID ]; +uint32_t vintf_cmdq_err_map[4 ]; } Tegra241CMDQV; +/* CMDQ-V Config page registers (offset 0x00000) */ +REG32(CONFIG, 0x0) +FIELD(CONFIG, CMDQV_EN, 0, 1) +FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3) +FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8) +FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8) +FIELD(CONFIG, CONS_DRAM_EN, 20, 1) + +REG32(PARAM, 0x4) +FIELD(PARAM, CMDQV_VER, 0, 4) +FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4) +FIELD(PARAM, CMDQV_NUM_VI_LOG2, 8, 4) +FIELD(PARAM, CMDQV_NUM_SID_PER_VI_LOG2, 12, 4) + +REG32(STATUS, 0x8) +FIELD (STATUS ,CMDQV_ENABLED ,0 ,1 ) + +/* SMMU_CMDQV_VI_ERR_MAP_0/1 definitions */ +#define A_VI_ERR_MAP_0 0x14 +#define A_VI_ERR_MAP_1 0x18 +#define V_VI_ERR_MAP_NO_ERROR (0) +#define V_VI_ERR_MAP_ERROR (1) + +/* SMMU_CMDQV_VI_INT_MASK_0/1 definitions */ +#define A_VI_INT_MASK_0 0x1c +#define A_VI_INT_MASK_1 0x20 +#define V_VI_INT_MASK_NOT_MASKED (0) +#define V_VI_INT_MASK_MASKED (1) + +/* SMMU_CMDQV_CMDQ_ERR_MAP_0-3 definitions */ +#define A_CMDQ_ERR_MAP_0 0x24 +#define A_CMDQ_ERR_MAP_1 0x28 +#define A_CMDQ_ERR_MAP_2 0x2c +#define A_CMDQ_ERR_MAP_3 0x30 + +/* +* CMDQ_ALLOC_MAP: one entry per physical VCMDQ. Hardware supports up to 128 +* entries (CMDQV_NUM_CMDQ_LOG2=7), but QEMU only exposes +* TEGRA241_CMDQV_MAX_CMDQ (=2) VCMDQs per VM so only entries 0 and 1 are +* defined here. +*/ +/* 2 identical register entries */ +#define SMMU_CMDQV_CMDQ_ALLOC_MAP_ (i) \ + REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4) \ + FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1) \ + FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7) \ + FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6) + +SMMU_CMDQV_CMDQ_ALLOC_MAP_(0) +SMMU_CMDQV_CMDQ_ALLOC_MAP_ (1 ) + +/* SMMU_CMDQV_VINTF0 registers (only VINTF0 is exposed to the guest) */ +REG32(VINTF0_CONFIG, 0x1000) +FIELD(VINTF0_CONFIG, ENABLE, 0, 1) +FIELD(VINTF0_CONFIG, VMID, 1, 16) +FIELD(VINTF0_CONFIG, HYP_OWN, 17, 1) + +REG32(VINTF0_STATUS, 0x1004) +FIELD(VINTF0_STATUS, ENABLE_OK, 0, 1) +FIELD(VINTF0_STATUS, STATUS, 1, 3) +FIELD(VINTF0_STATUS, VI_NUM_LVCMDQ, 16, 8) + +#define V_VINTF_STATUS_NO_ERROR (0 << 1) +#define V_VINTF_STATUS_VCMDQ_ERROR (1 << 1) + +/* +* SMMU_CMDQV_VINTF0_SID_MATCH/_REPLACE: 16 entries per VINTF +* (CMDQV_NUM_SID_PER_VI_LOG2=4). Only _0 and _15 are defined, +* used as switch case range bounds. +*/ +REG32(VINTF0_SID_MATCH_0, 0x1040) +FIELD(VINTF0_SID_MATCH_0, ENABLE, 0, 1) +FIELD(VINTF0_SID_MATCH_0, VIRT_SID, 1, 20) +#define A_VINTF0_SID_MATCH_15 (A_VINTF0_SID_MATCH_0 + 15 * 4) + +REG32(VINTF0_SID_REPLACE_0, 0x1080) +FIELD(VINTF0_SID_REPLACE_0, PHYS_SID, 0, 20) +#define A_VINTF0_SID_REPLACE_15 (A_VINTF0_SID_REPLACE_0 + 15 * 4) + +/* +* SMMU_CMDQV_VINTF0_LVCMDQ_ERR_MAP: 4 registers per VINTF covering 32 logical +* VCMDQs each. With TEGRA241_CMDQV_MAX_CMDQ=2, only MAP_0 bits [1:0] carry +* error state. MAP_1..MAP_3 always read as 0. Only _0 and _3 are defined, +* used as switch case range bounds. +*/ +REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) +FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32) +#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4) +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +184 -4
204 lines not on this page · raw message
hw/arm/trace-events +4 -0
@@ -72,6 +72,10 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x" +# tegra241-cmdqv +tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" + # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d" strongarm_ssp_read_underrun(void) "SSP rx underrun"
Signature · 1 line
2.43.0
Re: [PATCH v5 14/32] hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region
Quoted 373 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Tegra241 CMDQV exposes control and status registers in the CMDQ-V Config page (offset [
0x0,0x10000)) used to configure virtual command queue allocation and interrupt behavior.Add read/write emulation for the CMDQ-V Config region ([
CMDQV_BASE,CMDQV_CMDQ_BASE]), backed by a simple register cache. This includes CONFIG, PARAM, STATUS, VI error and interrupt maps, CMDQ allocation map and the VINTF0 related registers defined in the CMDQ-V Config space. Only VINTF0 is supported; VINTF1-63 are not.Dispatch writes on access size: Introduced
writel_mmiofor 4-byte andwritell_mmiofor 8-byte. Reads need no split as the MMIO framework masks the returned value to the access size.Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 110 +++++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 188 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 4 + 3 files changed, 298 insertions(+), 4 deletions(-)hw/arm/tegra241-cmdqv.h +110
-0@@ -10,10 +10,15 @@#ifndef HW_ARM_TEGRA241_CMDQV_H #define HW_ARM_TEGRA241_CMDQV_H +#include "hw/core/registerfields.h" +#define CMDQV_VER 1#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4 +#define TEGRA241_CMDQV_MAX_CMDQ (1U << CMDQV_NUM_CMDQ_LOG2) +#define TEGRA241_CMDQV_MAX_NUM_SID (1U << CMDQV_NUM_SID_PER_VI_LOG2) +/* * Tegra241 CMDQV MMIO layout (64KB pages) * @@ -31,8 +36,113 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; + +/* CMDQ-V Config page register cache */ +uint32_t config; +uint32_t param; +uint32_t status; +uint32_t vi_err_map[2 ]; +uint32_t vi_int_mask[2 ]; +uint32_t cmdq_err_map[4 ]; +uint32_t cmdq_alloc_map[TEGRA241_CMDQV_MAX_CMDQ ]; + +/* VINTF0 register cache (within CMDQ-V Config page) */ +uint32_t vintf_config; +uint32_t vintf_status; +uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID ]; +uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID ]; +uint32_t vintf_cmdq_err_map[4 ]; } Tegra241CMDQV; +/* CMDQ-V Config page registers (offset 0x00000) */ +REG32(CONFIG, 0x0) +FIELD(CONFIG, CMDQV_EN, 0, 1) +FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3) +FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8) +FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8) +FIELD(CONFIG, CONS_DRAM_EN, 20, 1) + +REG32(PARAM, 0x4) +FIELD(PARAM, CMDQV_VER, 0, 4) +FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4) +FIELD(PARAM, CMDQV_NUM_VI_LOG2, 8, 4) +FIELD(PARAM, CMDQV_NUM_SID_PER_VI_LOG2, 12, 4) + +REG32(STATUS, 0x8) +FIELD (STATUS ,CMDQV_ENABLED ,0 ,1 ) + +/* SMMU_CMDQV_VI_ERR_MAP_0/1 definitions */ +#define A_VI_ERR_MAP_0 0x14 +#define A_VI_ERR_MAP_1 0x18 +#define V_VI_ERR_MAP_NO_ERROR (0) +#define V_VI_ERR_MAP_ERROR (1) + +/* SMMU_CMDQV_VI_INT_MASK_0/1 definitions */ +#define A_VI_INT_MASK_0 0x1c +#define A_VI_INT_MASK_1 0x20 +#define V_VI_INT_MASK_NOT_MASKED (0) +#define V_VI_INT_MASK_MASKED (1) + +/* SMMU_CMDQV_CMDQ_ERR_MAP_0-3 definitions */ +#define A_CMDQ_ERR_MAP_0 0x24 +#define A_CMDQ_ERR_MAP_1 0x28 +#define A_CMDQ_ERR_MAP_2 0x2c +#define A_CMDQ_ERR_MAP_3 0x30 + +/* +* CMDQ_ALLOC_MAP: one entry per physical VCMDQ. Hardware supports up to 128 +* entries (CMDQV_NUM_CMDQ_LOG2=7), but QEMU only exposes +* TEGRA241_CMDQV_MAX_CMDQ (=2) VCMDQs per VM so only entries 0 and 1 are +* defined here. +*/ +/* 2 identical register entries */ +#define SMMU_CMDQV_CMDQ_ALLOC_MAP_ (i) \ + REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4) \ + FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1) \ + FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7) \ + FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6) + +SMMU_CMDQV_CMDQ_ALLOC_MAP_(0) +SMMU_CMDQV_CMDQ_ALLOC_MAP_ (1 ) + +/* SMMU_CMDQV_VINTF0 registers (only VINTF0 is exposed to the guest) */ +REG32(VINTF0_CONFIG, 0x1000) +FIELD(VINTF0_CONFIG, ENABLE, 0, 1) +FIELD(VINTF0_CONFIG, VMID, 1, 16) +FIELD(VINTF0_CONFIG, HYP_OWN, 17, 1) + +REG32(VINTF0_STATUS, 0x1004) +FIELD(VINTF0_STATUS, ENABLE_OK, 0, 1) +FIELD(VINTF0_STATUS, STATUS, 1, 3) +FIELD(VINTF0_STATUS, VI_NUM_LVCMDQ, 16, 8) + +#define V_VINTF_STATUS_NO_ERROR (0 << 1) +#define V_VINTF_STATUS_VCMDQ_ERROR (1 << 1) + +/* +* SMMU_CMDQV_VINTF0_SID_MATCH/_REPLACE: 16 entries per VINTF +* (CMDQV_NUM_SID_PER_VI_LOG2=4). Only _0 and _15 are defined, +* used as switch case range bounds. +*/ +REG32(VINTF0_SID_MATCH_0, 0x1040) +FIELD(VINTF0_SID_MATCH_0, ENABLE, 0, 1) +FIELD(VINTF0_SID_MATCH_0, VIRT_SID, 1, 20) +#define A_VINTF0_SID_MATCH_15 (A_VINTF0_SID_MATCH_0 + 15 * 4) + +REG32(VINTF0_SID_REPLACE_0, 0x1080) +FIELD(VINTF0_SID_REPLACE_0, PHYS_SID, 0, 20) +#define A_VINTF0_SID_REPLACE_15 (A_VINTF0_SID_REPLACE_0 + 15 * 4) + +/* +* SMMU_CMDQV_VINTF0_LVCMDQ_ERR_MAP: 4 registers per VINTF covering 32 logical +* VCMDQs each. With TEGRA241_CMDQV_MAX_CMDQ=2, only MAP_0 bits [1:0] carry +* error state. MAP_1..MAP_3 always read as 0. Only _0 and _3 are defined, +* used as switch case range bounds. +*/ +REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) +FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32) +#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4) +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +184
-4@@ -8,20 +8,200 @@*/ #include "qemu/osdep.h" +#include "qemu/log.h" #include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" +#include "trace.h" -static uint64_t tegra241_cmdqv_read_mmio (void *opaque,hwaddr offset,unsigned size) +static uint64_t tegra241_cmdqv_config_vintf_read (Tegra241CMDQV *cmdqv, +hwaddr offset) { -return 0 ; +int i; + +switch (offset) { +case A_VINTF0_CONFIG : +return cmdqv->vintf_config; +case A_VINTF0_STATUS : +return cmdqv->vintf_status; + case A_VINTF0_SID_MATCH_0 ... A_VINTF0_SID_MATCH_15: + i = (offset -A_VINTF0_SID_MATCH_0 ) /4 ; +return cmdqv->vintf_sid_match[i]; + case A_VINTF0_SID_REPLACE_0 ... A_VINTF0_SID_REPLACE_15: + i = (offset -A_VINTF0_SID_REPLACE_0 ) /4 ; +return cmdqv->vintf_sid_replace[i]; + case A_VINTF0_LVCMDQ_ERR_MAP_0 ... A_VINTF0_LVCMDQ_ERR_MAP_3: + i = (offset -A_VINTF0_LVCMDQ_ERR_MAP_0 ) /4 ; +return cmdqv->vintf_cmdq_err_map[i]; +default : +/* +* GLB_FILT_CFG_0 (offset 0xC) and GLB_FILT_DATA_0 (offset 0x10) are +* filter config and filter data registers. They are not required for +* normal VINTF operation and are not emulated. +*/ +qemu_log_mask (LOG_UNIMP ,"%s unhandled read access at 0x%" PRIx64"\n" , + __func__, offset); +return 0 ; + } +} + +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv, +hwaddr offset,uint64_t value) +{ +int i; + +switch (offset) { +case A_VINTF0_CONFIG : +/* +* Mask out HYP_OWN on guest writes. This bit selects Hypervisor (1) vs +* Guest (0) ownership of the CMDQ. Force it to 0 so the VINTF always +* remains guest-owned. +*/ + value &= ~R_VINTF0_CONFIG_HYP_OWN_MASK ; + + cmdqv->vintf_config = value; +if (value &R_VINTF0_CONFIG_ENABLE_MASK ) { + cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; + }else { + cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ; + } +break ; + case A_VINTF0_SID_MATCH_0 ... A_VINTF0_SID_MATCH_15: + i = (offset -A_VINTF0_SID_MATCH_0 ) /4 ; + cmdqv->vintf_sid_match[i] = value; +break ; + case A_VINTF0_SID_REPLACE_0 ... A_VINTF0_SID_REPLACE_15: + i = (offset -A_VINTF0_SID_REPLACE_0 ) /4 ; + cmdqv->vintf_sid_replace[i] = value; +break ; +default : +/* +* GLB_FILT_CFG_0 (offset 0xC) and GLB_FILT_DATA_0 (offset 0x10) are +* filter config and filter data registers. They are not required for +* normal VINTF operation and are not emulated. +*/ +qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset); +return ; + } } -static void tegra241_cmdqv_write_mmio (void *opaque,hwaddr offset,uint64_t value, -unsigned size) +static uint64_t tegra241_cmdqv_read_mmio (void *opaque,hwaddr offset, +unsigned size) { +Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque; +uint64_t val =0 ; + +if (offset >=TEGRA241_CMDQV_IO_LEN ) { +qemu_log_mask (LOG_UNIMP , +"%s offset 0x%" PRIx64" off limit (0x%x)\n" , __func__, + offset,TEGRA241_CMDQV_IO_LEN ); + goto out; + } + +switch (offset) { +case A_CONFIG : + val = cmdqv->config; +break ; +case A_PARAM : + val = cmdqv->param; +break ; +case A_STATUS : + val = cmdqv->status; +break ; + case A_VI_ERR_MAP_0 ... A_VI_ERR_MAP_1: + val = cmdqv->vi_err_map[(offset -A_VI_ERR_MAP_0 ) /4 ]; +break ; + case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1: + val = cmdqv->vi_int_mask[(offset -A_VI_INT_MASK_0 ) /4 ]; +break ; + case A_CMDQ_ERR_MAP_0 ... A_CMDQ_ERR_MAP_3: + val = cmdqv->cmdq_err_map[(offset -A_CMDQ_ERR_MAP_0 ) /4 ]; +break ; + case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: + val = cmdqv->cmdq_alloc_map[(offset -A_CMDQ_ALLOC_MAP_0 ) /4 ]; +break ; + case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3: + val =tegra241_cmdqv_config_vintf_read (cmdqv, offset); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled read access at 0x%" PRIx64"\n" , + __func__, offset); + } + +out: +trace_tegra241_cmdqv_read_mmio (offset, val, size); +return val; +} + +/* 4-byte MMIO write handler. */ +static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset, +uint32_t value) +{ +switch (offset) { +case A_CONFIG : + cmdqv->config = value; +if (value &R_CONFIG_CMDQV_EN_MASK ) { + cmdqv->status |=R_STATUS_CMDQV_ENABLED_MASK ; + }else { + cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK ; + } +break ; + case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1: + cmdqv->vi_int_mask[(offset -A_VI_INT_MASK_0 ) /4 ] = value; +break ; + case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: + cmdqv->cmdq_alloc_map[(offset -A_CMDQ_ALLOC_MAP_0 ) /4 ] = value; +break ; + case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3: +tegra241_cmdqv_config_vintf_write (cmdqv, offset, value); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset); + } +} + +/* +* 8-byte MMIO write handler. +*/ +static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset, +uint64_t value) +{ +qemu_log_mask (LOG_UNIMP , +"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , + __func__, offset); +} + +static void tegra241_cmdqv_write_mmio (void *opaque,hwaddr offset, +uint64_t value,unsigned size) +{ +Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque; + +if (offset >=TEGRA241_CMDQV_IO_LEN ) { +qemu_log_mask (LOG_UNIMP , +"%s offset 0x%" PRIx64" off limit (0x%x)\n" , __func__, + offset,TEGRA241_CMDQV_IO_LEN ); + goto out; + } + +switch (size) { +case 4 : +tegra241_cmdqv_writel_mmio (cmdqv, offset, value); +break ; +case 8 : +tegra241_cmdqv_writell_mmio (cmdqv, offset, value); +break ; +default : +qemu_log_mask (LOG_GUEST_ERROR , +"%s bad write size %u at 0x%" PRIx64"\n" , + __func__, size, offset); + } + +out: +trace_tegra241_cmdqv_write_mmio (offset, value, size); }static void tegra241_cmdqv_free_viommu (SMMUv3State *s)hw/arm/trace-events +0
-0
hw/arm/trace-events smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev · line 81
Quoted 10 lines
hw/arm/trace-events +4
-0@@ -72,6 +72,10 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x" +# tegra241-cmdqv +tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" + # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d" strongarm_ssp_read_underrun(void) "SSP rx underrun"
Eric
PATCH v5 15/32 hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Tegra241 CMDQV exposes per-VCMDQ register windows through two MMIO apertures:
Direct VCMDQ aperture (0x10000/0x20000): VCMDQ Page0/Page1 VINTF logical aperture (0x30000/0x40000): VINTF0 LVCMDQ Page0/Page1
Both apertures are hardware aliases of the same underlying registers:
Page 0 (control/status): CONS_INDX, PROD_INDX, CONFIG, STATUS,
GERROR, GERRORN
Page 1 (base/DRAM): BASE_L/H, CONS_INDX_BASE_DRAM_L/H
The direct aperture Page 0 is programmable at any time so long as CMDQV_EN is enabled. The VINTF (logical) aperture Page 0 is programmable only once SW has mapped a VCMDQ to a VINTF; the "logical" view is local to that VINTF.
Add read emulation for both apertures, backed by a single per-VCMDQ register cache. VINTF aperture reads are translated to their equivalent direct-aperture offset and served from the same cached state.
Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 reads will be served directly from the hardware-backed mmap'd page instead of the cache. Page 1 is also a hardware alias, but the kernel only exposes mmap for Page 0, so Page 1 reads always trap to QEMU and are served from cache.
--- hw/arm/tegra241-cmdqv.h | 215 ++++++++++++++++++++++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 95 ++++++++++++++++++ hw/arm/trace-events | 2 + 3 files changed, 312 insertions(+)
hw/arm/tegra241-cmdqv.h +215 -0
@@ -30,6 +30,13 @@*/ #define TEGRA241_CMDQV_IO_LEN 0x50000 +/* CMDQV MMIO aperture bases and VCMDQ stride */ +#define CMDQV_VCMDQ_PAGE0_BASE 0x10000/* CMDQV_CMDQ_BASE */ +#define CMDQV_VCMDQ_PAGE1_BASE 0x20000 +#define CMDQV_VINTF_PAGE0_BASE 0x30000/* CMDQV_VI_CMDQ_BASE */ +#define CMDQV_VINTF_PAGE1_BASE 0x40000 +#define CMDQV_VCMDQ_STRIDE 0x80 + typedef struct Tegra241CMDQV { struct iommu_viommu_tegra241_cmdqv cmdqv_data; SMMUv3AccelState *s_accel;
@@ -52,6 +59,19 @@ typedef struct Tegra241CMDQV {uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID ];uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID ];uint32_t vintf_cmdq_err_map[4 ]; +/* +* VCMDQ register cache. The direct (VCMDQ aperture) and logical +* (VINTF aperture) views are hardware aliases; both are served from +* this single cached copy. +*/ +uint32_t vcmdq_cons_indx[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_prod_indx[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_config[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_status[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_gerror[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_gerrorn[TEGRA241_CMDQV_MAX_CMDQ ]; +uint64_t vcmdq_base[TEGRA241_CMDQV_MAX_CMDQ ]; +uint64_t vcmdq_cons_indx_base[TEGRA241_CMDQV_MAX_CMDQ ]; } Tegra241CMDQV;/* CMDQ-V Config page registers (offset 0x00000) */
@@ -143,6 +163,201 @@ REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4) +/* +* VCMDQ Page 0 register window @ CMDQV_VCMDQ_PAGE0_BASE. +* Direct VCMDQ aperture; control and status registers. +*/ +#define SMMU_CMDQV_VCMDQi_CONS_INDX_ (i) \ + REG32(VCMDQ##i##_CONS_INDX, \ + CMDQV_VCMDQ_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20) \ + FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7) + +SMMU_CMDQV_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_(1) + +#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE 0 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE 1 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT 2 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS 4 + +#define SMMU_CMDQV_VCMDQi_PROD_INDX_ (i) \ + REG32(VCMDQ##i##_PROD_INDX, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20) + +SMMU_CMDQV_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VCMDQi_PROD_INDX_(1) + +#define SMMU_CMDQV_VCMDQi_CONFIG_ (i) \ + REG32(VCMDQ##i##_CONFIG, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1) + +SMMU_CMDQV_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VCMDQi_CONFIG_(1) + +#define SMMU_CMDQV_VCMDQi_STATUS_ (i) \ + REG32(VCMDQ##i##_STATUS, \ + CMDQV_VCMDQ_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1) + +SMMU_CMDQV_VCMDQi_STATUS_(0) +SMMU_CMDQV_VCMDQi_STATUS_(1) + +#define SMMU_CMDQV_VCMDQi_GERROR_ (i) \ + REG32(VCMDQ##i##_GERROR, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VCMDQi_GERROR_(0) +SMMU_CMDQV_VCMDQi_GERROR_(1) + +#define SMMU_CMDQV_VCMDQi_GERRORN_ (i) \ + REG32(VCMDQ##i##_GERRORN, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VCMDQi_GERRORN_(0) +SMMU_CMDQV_VCMDQi_GERRORN_ (1 ) + +/* +* VCMDQ Page 1 register window @ CMDQV_VCMDQ_PAGE1_BASE. +* Direct VCMDQ aperture; base and DRAM address registers. + */ +#define SMMU_CMDQV_VCMDQi_BASE_L_ (i) \ + REG32(VCMDQ##i##_BASE_L, CMDQV_VCMDQ_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5) \ + FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27) + +SMMU_CMDQV_VCMDQi_BASE_L_(0) +SMMU_CMDQV_VCMDQi_BASE_L_(1) + +#define SMMU_CMDQV_VCMDQi_BASE_H_ (i) \ + REG32(VCMDQ##i##_BASE_H, \ + CMDQV_VCMDQ_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16) + +SMMU_CMDQV_VCMDQi_BASE_H_(0) +SMMU_CMDQV_VCMDQi_BASE_H_(1) + +#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_ (i) \ + REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, \ + CMDQV_VCMDQ_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32) + +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) + +#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (i) \ + REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, \ + CMDQV_VCMDQ_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16) + +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (1 ) + +/* +* VINTF0 LVCMDQ Page 0 register window @ CMDQV_VINTF_PAGE0_BASE. +* Logical VCMDQ aperture (VCMDQs mapped via VINTF); control and +* status registers. Hardware alias of VCMDQ Page 0 above. + */ +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX, \ + CMDQV_VINTF_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20) \ + FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7) + +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(1) + +#define SMMU_CMDQV_VI_VCMDQi_PROD_INDX_ (i) \ + REG32(VI_VCMDQ##i##_PROD_INDX, \ + CMDQV_VINTF_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20) + +SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(1) + +#define SMMU_CMDQV_VI_VCMDQi_CONFIG_ (i) \ + REG32(VI_VCMDQ##i##_CONFIG, \ + CMDQV_VINTF_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1) + +SMMU_CMDQV_VI_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VI_VCMDQi_CONFIG_(1) + +#define SMMU_CMDQV_VI_VCMDQi_STATUS_ (i) \ + REG32(VI_VCMDQ##i##_STATUS, \ + CMDQV_VINTF_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1) + +SMMU_CMDQV_VI_VCMDQi_STATUS_(0) +SMMU_CMDQV_VI_VCMDQi_STATUS_(1) + +#define SMMU_CMDQV_VI_VCMDQi_GERROR_ (i) \ + REG32(VI_VCMDQ##i##_GERROR, \ + CMDQV_VINTF_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1) \ + FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VI_VCMDQi_GERROR_(0) +SMMU_CMDQV_VI_VCMDQi_GERROR_(1) + +#define SMMU_CMDQV_VI_VCMDQi_GERRORN_ (i) \ + REG32(VI_VCMDQ##i##_GERRORN, \ + CMDQV_VINTF_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VI_VCMDQi_GERRORN_(0) +SMMU_CMDQV_VI_VCMDQi_GERRORN_ (1 ) + +/* +* VINTF0 LVCMDQ Page 1 register window @ CMDQV_VINTF_PAGE1_BASE. +* Logical VCMDQ aperture (VCMDQs mapped via VINTF); base and DRAM +* address registers. Hardware alias of VCMDQ Page 1 above. + */ +#define SMMU_CMDQV_VI_VCMDQi_BASE_L_ (i) \ + REG32(VI_VCMDQ##i##_BASE_L, \ + CMDQV_VINTF_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5) \ + FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27) + +SMMU_CMDQV_VI_VCMDQi_BASE_L_(0) +SMMU_CMDQV_VI_VCMDQi_BASE_L_(1) + +#define SMMU_CMDQV_VI_VCMDQi_BASE_H_ (i) \ + REG32(VI_VCMDQ##i##_BASE_H, \ + CMDQV_VINTF_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16) + +SMMU_CMDQV_VI_VCMDQi_BASE_H_(0) +SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) + +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, \ + CMDQV_VINTF_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32) + +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) + +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, \ + CMDQV_VINTF_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16) + +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +95 -0
113 lines not on this page · raw message
hw/arm/trace-events +2 -0
@@ -75,6 +75,8 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
Re: [PATCH v5 15/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads
Quoted 133 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Tegra241 CMDQV exposes per-VCMDQ register windows through two MMIO apertures:
Direct VCMDQ aperture (0x10000/0x20000): VCMDQ Page0/Page1 VINTF logical aperture (0x30000/0x40000): VINTF0 LVCMDQ Page0/Page1
Both apertures are hardware aliases of the same underlying registers:
Page 0 (control/status):
CONS_INDX,PROD_INDX, CONFIG, STATUS,
GERROR, GERRORN
Page 1 (base/DRAM): BASE_L/H, CONS_INDX_BASE_DRAM_L/HThe direct aperture Page 0 is programmable at any time so long as
CMDQV_ENis enabled. The VINTF (logical) aperture Page 0 is programmable only once SW has mapped a VCMDQ to a VINTF; the "logical" view is local to that VINTF.Add read emulation for both apertures, backed by a single per-VCMDQ register cache. VINTF aperture reads are translated to their equivalent direct-aperture offset and served from the same cached state.
Once
IOMMU_HW_QUEUE_ALLOCandviommu_mmapare wired up in a subsequent patch, Page 0 reads will be served directly from the hardware-backed mmap'd page instead of the cache. Page 1 is also a hardware alias, but the kernel only exposes mmap for Page 0, so Page 1 reads always trap to QEMU and are served from cache.Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 215 ++++++++++++++++++++++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 95 ++++++++++++++++++ hw/arm/trace-events | 2 + 3 files changed, 312 insertions(+)hw/arm/tegra241-cmdqv.h +72
-0@@ -30,6 +30,13 @@*/ #define TEGRA241_CMDQV_IO_LEN 0x50000 +/* CMDQV MMIO aperture bases and VCMDQ stride */ +#define CMDQV_VCMDQ_PAGE0_BASE 0x10000/* CMDQV_CMDQ_BASE */ +#define CMDQV_VCMDQ_PAGE1_BASE 0x20000 +#define CMDQV_VINTF_PAGE0_BASE 0x30000/* CMDQV_VI_CMDQ_BASE */ +#define CMDQV_VINTF_PAGE1_BASE 0x40000 +#define CMDQV_VCMDQ_STRIDE 0x80 + typedef struct Tegra241CMDQV { struct iommu_viommu_tegra241_cmdqv cmdqv_data; SMMUv3AccelState *s_accel;@@ -52,6 +59,19 @@ typedef struct Tegra241CMDQV {uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID ];uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID ];uint32_t vintf_cmdq_err_map[4 ]; +/* +* VCMDQ register cache. The direct (VCMDQ aperture) and logical +* (VINTF aperture) views are hardware aliases; both are served from +* this single cached copy. +*/ +uint32_t vcmdq_cons_indx[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_prod_indx[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_config[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_status[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_gerror[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_gerrorn[TEGRA241_CMDQV_MAX_CMDQ ]; +uint64_t vcmdq_base[TEGRA241_CMDQV_MAX_CMDQ ]; +uint64_t vcmdq_cons_indx_base[TEGRA241_CMDQV_MAX_CMDQ ]; } Tegra241CMDQV;/* CMDQ-V Config page registers (offset 0x00000) */ @@ -143,6 +163,201 @@ REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4) +/* +* VCMDQ Page 0 register window @ CMDQV_VCMDQ_PAGE0_BASE. +* Direct VCMDQ aperture; control and status registers. +*/ +#define SMMU_CMDQV_VCMDQi_CONS_INDX_ (i) \ + REG32(VCMDQ##i##_CONS_INDX, \ + CMDQV_VCMDQ_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20) \ + FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7) + +SMMU_CMDQV_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_(1) + +#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE 0 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE 1 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT 2 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS 4 + +#define SMMU_CMDQV_VCMDQi_PROD_INDX_ (i) \ + REG32(VCMDQ##i##_PROD_INDX, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20) + +SMMU_CMDQV_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VCMDQi_PROD_INDX_(1) + +#define SMMU_CMDQV_VCMDQi_CONFIG_ (i) \ + REG32(VCMDQ##i##_CONFIG, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1) + +SMMU_CMDQV_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VCMDQi_CONFIG_(1) + +#define SMMU_CMDQV_VCMDQi_STATUS_ (i) \ + REG32(VCMDQ##i##_STATUS, \ + CMDQV_VCMDQ_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1) + +SMMU_CMDQV_VCMDQi_STATUS_(0) +SMMU_CMDQV_VCMDQi_STATUS_(1) + +#define SMMU_CMDQV_VCMDQi_GERROR_ (i) \ + REG32(VCMDQ##i##_GERROR, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VCMDQi_GERROR_(0) +SMMU_CMDQV_VCMDQi_GERROR_ (1 )
hw/arm/tegra241-cmdqv.h REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) · line 227
Quoted 10 lines
hw/arm/tegra241-cmdqv.h +10
-0+ +#define SMMU_CMDQV_VCMDQi_GERRORN_ (i) \ + REG32(VCMDQ##i##_GERRORN, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VCMDQi_GERRORN_(0) +SMMU_CMDQV_VCMDQi_GERRORN_ (1 )
nit, what I meant in my v4 comments is I would prefer we group the eventual register definition to represent the actual layout of each page, instead of interleaving them with actual macros:
This would look like:
/* page 0 */
/* vcmdq0 */
+SMMU_CMDQV_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(0)
/* vcmdq1 */
+SMMU_CMDQV_VCMDQi_CONS_INDX_(1)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(1)
+SMMU_CMDQV_VCMDQi_CONFIG_(1)
+SMMU_CMDQV_VCMDQi_STATUS_(1)
+SMMU_CMDQV_VCMDQi_GERROR_(1)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)
same for page 1
To me it looks easier to remember the layout without looking at the spec.
Taste & colors, what do you prefer?
Quoted 261 lines
hw/arm/tegra241-cmdqv.h +133
-0+ +/* +* VCMDQ Page 1 register window @ CMDQV_VCMDQ_PAGE1_BASE. +* Direct VCMDQ aperture; base and DRAM address registers. +*/ +#define SMMU_CMDQV_VCMDQi_BASE_L_ (i) \ + REG32(VCMDQ##i##_BASE_L, CMDQV_VCMDQ_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5) \ + FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27) + +SMMU_CMDQV_VCMDQi_BASE_L_(0) +SMMU_CMDQV_VCMDQi_BASE_L_(1) + +#define SMMU_CMDQV_VCMDQi_BASE_H_ (i) \ + REG32(VCMDQ##i##_BASE_H, \ + CMDQV_VCMDQ_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16) + +SMMU_CMDQV_VCMDQi_BASE_H_(0) +SMMU_CMDQV_VCMDQi_BASE_H_(1) + +#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_ (i) \ + REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, \ + CMDQV_VCMDQ_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32) + +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) + +#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (i) \ + REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, \ + CMDQV_VCMDQ_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16) + +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (1 ) + +/* +* VINTF0 LVCMDQ Page 0 register window @ CMDQV_VINTF_PAGE0_BASE. +* Logical VCMDQ aperture (VCMDQs mapped via VINTF); control and +* status registers. Hardware alias of VCMDQ Page 0 above. + */ +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX, \ + CMDQV_VINTF_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20) \ + FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7) + +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(1) + +#define SMMU_CMDQV_VI_VCMDQi_PROD_INDX_ (i) \ + REG32(VI_VCMDQ##i##_PROD_INDX, \ + CMDQV_VINTF_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20) + +SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(1) + +#define SMMU_CMDQV_VI_VCMDQi_CONFIG_ (i) \ + REG32(VI_VCMDQ##i##_CONFIG, \ + CMDQV_VINTF_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1) + +SMMU_CMDQV_VI_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VI_VCMDQi_CONFIG_(1) + +#define SMMU_CMDQV_VI_VCMDQi_STATUS_ (i) \ + REG32(VI_VCMDQ##i##_STATUS, \ + CMDQV_VINTF_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1) + +SMMU_CMDQV_VI_VCMDQi_STATUS_(0) +SMMU_CMDQV_VI_VCMDQi_STATUS_(1) + +#define SMMU_CMDQV_VI_VCMDQi_GERROR_ (i) \ + REG32(VI_VCMDQ##i##_GERROR, \ + CMDQV_VINTF_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1) \ + FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VI_VCMDQi_GERROR_(0) +SMMU_CMDQV_VI_VCMDQi_GERROR_(1) + +#define SMMU_CMDQV_VI_VCMDQi_GERRORN_ (i) \ + REG32(VI_VCMDQ##i##_GERRORN, \ + CMDQV_VINTF_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VI_VCMDQi_GERRORN_(0) +SMMU_CMDQV_VI_VCMDQi_GERRORN_ (1 ) + +/* +* VINTF0 LVCMDQ Page 1 register window @ CMDQV_VINTF_PAGE1_BASE. +* Logical VCMDQ aperture (VCMDQs mapped via VINTF); base and DRAM +* address registers. Hardware alias of VCMDQ Page 1 above. + */ +#define SMMU_CMDQV_VI_VCMDQi_BASE_L_ (i) \ + REG32(VI_VCMDQ##i##_BASE_L, \ + CMDQV_VINTF_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5) \ + FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27) + +SMMU_CMDQV_VI_VCMDQi_BASE_L_(0) +SMMU_CMDQV_VI_VCMDQi_BASE_L_(1) + +#define SMMU_CMDQV_VI_VCMDQi_BASE_H_ (i) \ + REG32(VI_VCMDQ##i##_BASE_H, \ + CMDQV_VINTF_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16) + +SMMU_CMDQV_VI_VCMDQi_BASE_H_(0) +SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) + +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, \ + CMDQV_VINTF_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32) + +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) + +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, \ + CMDQV_VINTF_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16) + +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +95
-0@@ -16,6 +16,75 @@#include "tegra241-cmdqv.h" #include "trace.h" +/* +* Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +*/ +static uint64_t tegra241_cmdqv_read_vcmdq_page0 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index) +{ +uint64_t val =0 ; + +switch (offset0) { +case A_VCMDQ0_CONS_INDX : + val = cmdqv->vcmdq_cons_indx[index]; +break ; +case A_VCMDQ0_PROD_INDX : + val = cmdqv->vcmdq_prod_indx[index]; +break ; +case A_VCMDQ0_CONFIG : + val = cmdqv->vcmdq_config[index]; +break ; +case A_VCMDQ0_STATUS : + val = cmdqv->vcmdq_status[index]; +break ; +case A_VCMDQ0_GERROR : + val = cmdqv->vcmdq_gerror[index]; +break ; +case A_VCMDQ0_GERRORN : + val = cmdqv->vcmdq_gerrorn[index]; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled read access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_read_vcmdq_page0 (index, offset0, val); +return val; +} + +/* +* Read a VCMDQ Page 1 register (base / DRAM address) using VCMDQ0_* offsets. +*/ +static uint64_t tegra241_cmdqv_read_vcmdq_page1 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index) +{ +uint64_t val =0 ; + +switch (offset0) { +case A_VCMDQ0_BASE_L : + val = cmdqv->vcmdq_base[index]; +break ; +case A_VCMDQ0_BASE_H : + val = cmdqv->vcmdq_base[index] >>32 ; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + val = cmdqv->vcmdq_cons_indx_base[index]; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_H : + val = cmdqv->vcmdq_cons_indx_base[index] >>32 ; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled read access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_read_vcmdq_page1 (index, offset0, val); +return val; +} +static uint64_t tegra241_cmdqv_config_vintf_read (Tegra241CMDQV *cmdqv,hwaddr offset) {@@ -93,6 +162,7 @@ static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset, {Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;uint64_t val =0 ; +int index;if (offset >=TEGRA241_CMDQV_IO_LEN ) { qemu_log_mask(LOG_UNIMP,@@ -126,6 +196,31 @@ static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset, case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3: val =tegra241_cmdqv_config_vintf_read (cmdqv, offset);break ; + case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN: +/* +* VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers. +* Translate the VINTF aperture offset to its VCMDQ Page0 equivalent +* and fall through to the Page0 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE0_BASE -CMDQV_VCMDQ_PAGE0_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN: +/* +* Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a +* CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize +* to the VCMDQ0_* offset before calling the Page 0 helper. +*/ + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +return tegra241_cmdqv_read_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index); + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +return tegra241_cmdqv_read_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index);default :qemu_log_mask (LOG_UNIMP ,"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset);hw/arm/trace-events +0
-0
hw/arm/trace-events smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS · line 82
Quoted 8 lines
hw/arm/trace-events +2
-0@@ -75,6 +75,8 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Besides looks good to me
Eric
RE: [PATCH v5 15/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads
Quoted 15 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 06:49
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 15/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads
[...]
Quoted 37 lines
hw/arm/tegra241-cmdqv.h REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
Quoted 3 lines
hw/arm/tegra241-cmdqv.h +3
-0+#define SMMU_CMDQV_VCMDQi_GERRORN_ (i) \ + REG32(VCMDQ##i##_GERRORN, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE)\
hw/arm/tegra241-cmdqv.h REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) · line 227
Quoted 6 lines
hw/arm/tegra241-cmdqv.h +6
-0+ FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +SMMU_CMDQV_VCMDQi_GERRORN_(0) +SMMU_CMDQV_VCMDQi_GERRORN_ (1 )nit, what I meant in my v4 comments is I would prefer we group the eventual register definition to represent the actual layout of each page, instead of interleaving them with actual macros:
This would look like:/* page 0 */
/* vcmdq0 */
hw/arm/tegra241-cmdqv.h +6
-0+SMMU_CMDQV_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VCMDQi_STATUS_(0) +SMMU_CMDQV_VCMDQi_GERROR_(0) +SMMU_CMDQV_VCMDQi_GERRORN_ (0 )/* vcmdq1 */
hw/arm/tegra241-cmdqv.h +6
-0+SMMU_CMDQV_VCMDQi_CONS_INDX_(1) +SMMU_CMDQV_VCMDQi_PROD_INDX_(1) +SMMU_CMDQV_VCMDQi_CONFIG_(1) +SMMU_CMDQV_VCMDQi_STATUS_(1) +SMMU_CMDQV_VCMDQi_GERROR_(1) +SMMU_CMDQV_VCMDQi_GERRORN_ (1 )
Quoted 6 lines
same for page 1
To me it looks easier to remember the layout without looking at the spec.
Taste & colors, what do you prefer?
Looks like I misunderstood your v4 comment. Sure, I will switch to this layout.
Thanks,
Shameer
PATCH v5 16/32 hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache. Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.
Ignore VCMDQ BASE writes if the VCMDQ is already enabled.
--- hw/arm/tegra241-cmdqv.c | 164 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 2 + 2 files changed, 164 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +162 -2
@@ -16,6 +16,16 @@#include "tegra241-cmdqv.h" #include "trace.h" +/* +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +*/ +static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t status = cmdqv->vcmdq_status[index]; + +return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. *
@@ -116,6 +126,108 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, } } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/ +static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value) +{ +switch (offset0) { +case A_VCMDQ0_CONS_INDX : + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : + cmdqv->vcmdq_prod_indx[index] = value; +break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }else { + cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : + cmdqv->vcmdq_gerrorn[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_H : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],32 ,32 , value); +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L. +*/ +static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint64_t value) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = value; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled 64-bit write at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) {
@@ -235,6 +347,8 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +int index; +switch (offset) {case A_CONFIG : cmdqv->config = value;
@@ -253,6 +367,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value);break ; + case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN: +/* +* VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers. +* Translate the VINTF aperture offset to its VCMDQ Page0 equivalent +* and fall through to the Page0 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE0_BASE -CMDQV_VCMDQ_PAGE0_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN: +/* +* Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a +* CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize +* to the VCMDQ0_* offset before calling the Page 0 helper. +*/ + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ; + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ;default :qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset);
@@ -260,14 +401,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, }/* -* 8-byte MMIO write handler. +* 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept +* full 64-bit writes; other offsets are write-ignored. */ static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { -qemu_log_mask (LOG_UNIMP , +int index; + +switch (offset) { + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* +* VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers. +* Translate the VINTF aperture offset to its VCMDQ Page1 equivalent +* and fall through to the Page1 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +2 -0
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 238 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE,
CONS_INDX_BASE) always update the cache. OnceIOMMU_HW_QUEUE_ALLOCandviommu_mmapare wired up in a subsequent patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.Ignore VCMDQ BASE writes if the VCMDQ is already enabled.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 164 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 2 + 2 files changed, 164 insertions(+), 2 deletions(-)hw/arm/tegra241-cmdqv.c +162
-2@@ -16,6 +16,16 @@#include "tegra241-cmdqv.h" #include "trace.h" +/* +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +*/ +static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t status = cmdqv->vcmdq_status[index]; + +return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. * @@ -116,6 +126,108 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, } } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/ +static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value) +{ +switch (offset0) { +case A_VCMDQ0_CONS_INDX : + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : + cmdqv->vcmdq_prod_indx[index] = value; +break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }else { + cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : + cmdqv->vcmdq_gerrorn[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_H : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],32 ,32 , value); +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L. +*/ +static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint64_t value) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = value; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled 64-bit write at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) {@@ -235,6 +347,8 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +int index; +switch (offset) {case A_CONFIG : cmdqv->config = value;@@ -253,6 +367,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value);break ; + case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN: +/* +* VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers. +* Translate the VINTF aperture offset to its VCMDQ Page0 equivalent +* and fall through to the Page0 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE0_BASE -CMDQV_VCMDQ_PAGE0_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN: +/* +* Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a +* CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize +* to the VCMDQ0_* offset before calling the Page 0 helper. +*/ + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ; + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ;default :qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset);@@ -260,14 +401,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, }/* -* 8-byte MMIO write handler. +* 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept +* full 64-bit writes; other offsets are write-ignored. */ static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { -qemu_log_mask (LOG_UNIMP , +int index; + +switch (offset) { + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* +* VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers. +* Translate the VINTF aperture offset to its VCMDQ Page1 equivalent +* and fall through to the Page1 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,hw/arm/trace-events +0
-0
hw/arm/trace-events tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: · line 81
Quoted 5 lines
hw/arm/trace-events +2
-0@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
actually would it be possible to improve the trace points by adding the info about whether the VI or the global page is accessed. This would allow to discriminate both. Same for read
Eric
Quoted 3 lines
hw/arm/trace-events +0
-0# strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
RE: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 14 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 06:56
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
[...]
Quoted 7 lines
Quoted 1 line
+tegra241_cmdqv_write_vcmdq_page0(int index,
uint64_toffset0,uint64_tval) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Quoted 1 line
+tegra241_cmdqv_write_vcmdq_page1(int index,
uint64_toffset0,uint64_tval) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 actually would it be possible to improve the trace points by adding the info about whether the VI or the global page is accessed. This would allow to discriminate both. Same for read
Ok. Will add for v6.
Thanks,
Shameer
Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
In reply to Shameer KolothumHi Shameer,
Quoted 58 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE,
CONS_INDX_BASE) always update the cache. OnceIOMMU_HW_QUEUE_ALLOCandviommu_mmapare wired up in a subsequent patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.Ignore VCMDQ BASE writes if the VCMDQ is already enabled.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 164 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 2 + 2 files changed, 164 insertions(+), 2 deletions(-)hw/arm/tegra241-cmdqv.c +19
-0@@ -16,6 +16,16 @@#include "tegra241-cmdqv.h" #include "trace.h" +/* +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +*/ +static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t status = cmdqv->vcmdq_status[index]; + +return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. * @@ -116,6 +126,108 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, } } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 147
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value) +{ +switch (offset0) { +case A_VCMDQ0_CONS_INDX : + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : + cmdqv->vcmdq_prod_indx[index] = value;
so there is no cmd consumption triggered by an increment of the producer index as opposed to std smmu emulation. This shall at least be explained in the commit msg and maybe in the code, + the justification. It is a bit strange to emulate the MMIO accesses while not doing anything functional out of them.
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 158
Quoted 11 lines
hw/arm/tegra241-cmdqv.c +11
-0+break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }else { + cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : + cmdqv->vcmdq_gerrorn[index] = value;
shouldn't it do something functional besides storing the value on std SMMU we resume consuming cmds. interaction with interrupts, CONS_INDX?
Quoted 16 lines
hw/arm/tegra241-cmdqv.c +16
-0+break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value) +{
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 184
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+switch (offset0) { +case A_VCMDQ0_BASE_L : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : +if (tegra241_vcmdq_enabled (cmdqv, index)) {
where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting
Quoted 133 lines
hw/arm/tegra241-cmdqv.c +96
-2+return ; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_H : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],32 ,32 , value); +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L. +*/ +static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint64_t value) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : +if (tegra241_vcmdq_enabled (cmdqv, index)) { +return ; + } + cmdqv->vcmdq_base[index] = value; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled 64-bit write at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) {@@ -235,6 +347,8 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +int index; +switch (offset) {case A_CONFIG : cmdqv->config = value;@@ -253,6 +367,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value);break ; + case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN: +/* +* VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers. +* Translate the VINTF aperture offset to its VCMDQ Page0 equivalent +* and fall through to the Page0 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE0_BASE -CMDQV_VCMDQ_PAGE0_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN: +/* +* Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a +* CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize +* to the VCMDQ0_* offset before calling the Page 0 helper. +*/ + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ; + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ;default :qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset);@@ -260,14 +401,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, }/* -* 8-byte MMIO write handler. +* 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept +* full 64-bit writes; other offsets are write-ignored. */ static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { -qemu_log_mask (LOG_UNIMP , +int index; + +switch (offset) { + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* +* VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers. +* Translate the VINTF aperture offset to its VCMDQ Page1 equivalent +* and fall through to the Page1 dispatch below. +*/ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; +QEMU_FALLTHROUGH ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,hw/arm/trace-events +0
-0
hw/arm/trace-events tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: · line 84
Quoted 8 lines
hw/arm/trace-events +2
-0@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Thanks
Eric
RE: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 93 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 08:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writesExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 11 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE,
CONS_INDX_BASE) always update the cache. OnceIOMMU_HW_QUEUE_ALLOCandviommu_mmapare wired up in asubsequent
Quoted 10 lines
patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.
Ignore VCMDQ BASE writes if the VCMDQ is already enabled.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 164+++++++++++++++++++++++++++++++++++++++-
hw/arm/tegra241-cmdqv.c static uint64_t
Quoted 25 lines
hw/arm/trace-events | 2 + 2 files changed, 164 insertions(+), 2 deletions(-)hw/arm/tegra241-cmdqv.c +10
-0@@ -16,6 +16,16 @@#include "tegra241-cmdqv.h" #include "trace.h" +/* +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +*/ +static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t status = cmdqv->vcmdq_status[index]; + +return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. * @@ -116,6 +126,108 @@ static uint64_ttegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 13 lines
hw/arm/tegra241-cmdqv.c +10
-0} } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/ +static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV*cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 147
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9
-0+ hwaddr offset0, int index, + uint32_t value) +{ + switch (offset0) { + case A_VCMDQ0_CONS_INDX: + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : + cmdqv->vcmdq_prod_indx[index] = value;
Quoted 5 lines
so there is no cmd consumption triggered by an increment of the producer index as opposed to std smmu emulation. This shall at least be explained in the commit msg and maybe in the code, + the justification. It is a bit strange to emulate the MMIO accesses while not doing anything functional out of them.
There is indeed a difference here w.r.t. std SMMU emulation. CMDQV
is a hardware-accelerated feature, and there isn't much value in emulating
the queue functionality without the HW backing.
From Spec, p.176:
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU"
So no actual command processing can happen until the VCMDQ is allocated to a Virtual Interface. Until then, reads/writes only update the register cache.
I will make this clear in the commit log.
Quoted 8 lines
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c static uint64_t · line 153
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+ }else { + cmdqv->vcmdq_status[index] &=~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
Quoted 8 lines
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 158
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+ } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : + cmdqv->vcmdq_gerrorn[index] = value;shouldn't it do something functional besides storing the value on std SMMU we resume consuming cmds. interaction with interrupts,
CONS_INDX?
Same rationale as above. In the pre-allocation state there is no real queue function to drive, so there is nothing functional for QEMU to do on a GERRORN write.
Quoted 27 lines
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 13 lines
hw/arm/tegra241-cmdqv.c +13
-0+break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV*cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 184
Quoted 13 lines
hw/arm/tegra241-cmdqv.c +13
-0+ hwaddr offset0, int index, + uint32_t value) +{ + switch (offset0) { + case A_VCMDQ0_BASE_L: + if (tegra241_vcmdq_enabled(cmdqv, index)) { + return; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : +if (tegra241_vcmdq_enabled (cmdqv, index)) {
Quoted 4 lines
where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting
Right. I don’t think it is explicitly there in the spec.
However, Nicolin mentioned about such a restriction during v4 discussion:
https://lore.kernel.org/qemu-devel/[email protected]/
AFAICS, it looks like an implied assumption based on the "Enabling the virtual CMDQ" sequence or I missed it.
I will double check.
Thanks,
Shameer
Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 108 lines · Shameer Kolothum Thodi
Quoted 92 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 08:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writesExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 11 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE,
CONS_INDX_BASE) always update the cache. OnceIOMMU_HW_QUEUE_ALLOCandviommu_mmapare wired up in asubsequent
Quoted 10 lines
patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.
Ignore VCMDQ BASE writes if the VCMDQ is already enabled.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 164+++++++++++++++++++++++++++++++++++++++-
hw/arm/tegra241-cmdqv.c static uint64_t
Quoted 25 lines
hw/arm/trace-events | 2 + 2 files changed, 164 insertions(+), 2 deletions(-)hw/arm/tegra241-cmdqv.c +10
-0@@ -16,6 +16,16 @@#include "tegra241-cmdqv.h" #include "trace.h" +/* +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +*/ +static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t status = cmdqv->vcmdq_status[index]; + +return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. * @@ -116,6 +126,108 @@ static uint64_ttegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 13 lines
hw/arm/tegra241-cmdqv.c +10
-0} } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/ +static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV*cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 147
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9
-0+ hwaddr offset0, int index, + uint32_t value) +{ + switch (offset0) { + case A_VCMDQ0_CONS_INDX: + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : + cmdqv->vcmdq_prod_indx[index] = value;Quoted 5 lines
so there is no cmd consumption triggered by an increment of the producer index as opposed to std smmu emulation. This shall at least be explained in the commit msg and maybe in the code, + the justification. It is a bit strange to emulate the MMIO accesses while not doing anything functional out of them.
There is indeed a difference here w.r.t. std SMMU emulation. CMDQV
is a hardware-accelerated feature, and there isn't much value in emulating
the queue functionality without the HW backing.From Spec, p.176:
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU"
Quoted 4 lines
So no actual command processing can happen until the VCMDQ is allocated to a Virtual Interface. Until then, reads/writes only update the register cache.
I will make this clear in the commit log.
Yes this helps
Quoted 14 lines
Quoted 14 lines
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +4
-0+break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c static uint64_t · line 153
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+ }else { + cmdqv->vcmdq_status[index] &=~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+ } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : + cmdqv->vcmdq_gerrorn[index] = value;
Quoted 6 lines
Quoted 3 lines
shouldn't it do something functional besides storing the value on std SMMU we resume consuming cmds. interaction with interrupts,
CONS_INDX?Same rationale as above. In the pre-allocation state there is no real queue function to drive, so there is nothing functional for QEMU to do on a GERRORN write.
also I would add a comment in the code
Quoted 34 lines
Quoted 28 lines
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +3
-0+break ; +default : + qemu_log_mask(LOG_UNIMP,hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV*cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 184
Quoted 13 lines
hw/arm/tegra241-cmdqv.c +13
-0+ hwaddr offset0, int index, + uint32_t value) +{ + switch (offset0) { + case A_VCMDQ0_BASE_L: + if (tegra241_vcmdq_enabled(cmdqv, index)) { + return; + } + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : +if (tegra241_vcmdq_enabled (cmdqv, index)) {Quoted 4 lines
where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting
Right. I don’t think it is explicitly there in the spec.
Quoted 7 lines
However, Nicolin mentioned about such a restriction during v4 discussion:
https://lore.kernel.org/qemu-devel/[email protected]/AFAICS, it looks like an implied assumption based on the "Enabling the virtual CMDQ" sequence or I missed it.
I will double check.
OK
Thanks
Eric
Quoted 3 lines
Thanks,
Shameer
RE: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 14 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:08
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
[...]
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ case A_VCMDQ0_BASE_H: + if (tegra241_vcmdq_enabled(cmdqv, index)) {where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting
Right. I don’t think it is explicitly there in the spec.
Quoted 8 lines
Quoted 7 lines
However, Nicolin mentioned about such a restriction during v4 discussion:
https://lore.kernel.org/qemu-devel/[email protected]/AFAICS, it looks like an implied assumption based on the "Enabling the virtual CMDQ" sequence or I missed it.
I will double check.
OK
I did a hack to the host kernel driver to see how the HW behaves if we reprogram the BASE after the VCMDQ is enabled:
+ /* HACK: re-write BASE after VCMDQ_EN */
+ {
+ u64 orig = vcmdq->cmdq.q.q_base;
+
+ printk("%s: Shameer: re writing base after vcmdq enable..\n", __func__);
+ writeq_relaxed(0, REG_VCMDQ_PAGE1(vcmdq, BASE));
+ pr_info("Base : wrote 0, readback=0x%llx GERROR=0x%x STATUS=0x%x\n",
+ readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)),
+ readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, GERROR)),
+ readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, STATUS)));
+
+ /* Put it back before something tries to use it */
+ writeq_relaxed(orig, REG_VCMDQ_PAGE1(vcmdq, BASE));
+ pr_info("base: restored, readback=0x%llx\n",+ readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)));
+ }
+
From the boot logs, the HW accepted the BASE rewrite:
[ 1.504735] tegra241_vcmdq_hw_init: Shameer: re writing base after vcmdq enable.. [ 1.504737] Base : wrote 0, readback=0x0 GERROR=0x0 STATUS=0x1 [ 1.504739] base: restored, readback=0xfa000013
So writes to BASE after enable are not ignored at the register level. But I don't think command processing will work if we change BASE on a live queue.
From the QEMU side, the current setup_vcmdq() frees the existing hw_queue before calling IOMMU_HW_QUEUE_ALLOC again, otherwise the kernel returns -EEXIST. So if we allow BASE rewrite while the VCMDQ is enabled, it will trigger a destroy+alloc cycle in the host driver.
I tried removing the tegra241_vcmdq_enabled(cmdqv, index) restriction and rewriting BASE triggering a setup_vcmdq() again, and the guest ended up with:
arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000023 [hwprod 0x00000023, hwcons 0x00000000] arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000025 [hwprod 0x00000025, hwcons 0x00000000]
I think it is probably better to keep this behaviour (let the rewrite go through and surface as a CMD_SYNC timeout) than to silently ignore the BASE reprogram.
@Nicolin, thoughts?
Thanks,
Shameer
Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 40 lines · Shameer Kolothum Thodi
I did a hack to the host kernel driver to see how the HW behaves if we reprogram the BASE after the VCMDQ is enabled:
(quoted diff) +17
-0+ /* HACK: re-write BASE after VCMDQ_EN */ + { + u64 orig = vcmdq->cmdq.q.q_base; + + printk("%s: Shameer: re writing base after vcmdq enable..\n", __func__); + writeq_relaxed(0, REG_VCMDQ_PAGE1(vcmdq, BASE)); + pr_info("Base : wrote 0, readback=0x%llx GERROR=0x%x STATUS=0x%x\n", + readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)), + readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, GERROR)), + readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, STATUS))); + + /* Put it back before something tries to use it */ + writeq_relaxed(orig, REG_VCMDQ_PAGE1(vcmdq, BASE)); + pr_info("base: restored, readback=0x%llx\n", + readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE))); + } +From the boot logs, the HW accepted the BASE rewrite:
[ 1.504735] tegra241_vcmdq_hw_init: Shameer: re writing base after vcmdq enable.. [ 1.504737] Base : wrote 0, readback=0x0 GERROR=0x0 STATUS=0x1 [ 1.504739] base: restored, readback=0xfa000013So writes to BASE after enable are not ignored at the register level. But I don't think command processing will work if we change BASE on a live queue.
From the QEMU side, the current
setup_vcmdq()frees the existinghw_queuebefore callingIOMMU_HW_QUEUE_ALLOCagain, otherwise the kernel returns-EEXIST. So if we allow BASE rewrite while the VCMDQ is enabled, it will trigger a destroy+alloc cycle in the host driver.I tried removing the
tegra241_vcmdq_enabled(cmdqv, index)restriction and rewriting BASE triggering asetup_vcmdq()again, and the guest ended up with:
Quoted 8 lines
arm-smmu-v3 arm-smmu-v3.0.auto:
CMD_SYNCtimeout at0x00000023[hwprod0x00000023, hwcons0x00000000] arm-smmu-v3 arm-smmu-v3.0.auto:CMD_SYNCtimeout at0x00000025[hwprod0x00000025, hwcons0x00000000]I think it is probably better to keep this behaviour (let the rewrite go through and surface as a
CMD_SYNCtimeout) than to silently ignore the BASE reprogram.@Nicolin, thoughts?
You're asking about guest writing the direct MMIO space (trapped):VCMDQ_EN and then BASE?
If so, yes.
Nicolin
PATCH v5 17/32 hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.
The kernel provides an mmap offset for the VINTF Page0 region during vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.
When VINTF is enabled, mmap the kernel provided Page0 region and set ENABLE_OK only if the mmap succeeds. Unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation support.
--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
@@ -37,12 +37,15 @@#define CMDQV_VINTF_PAGE1_BASE 0x40000#define CMDQV_VCMDQ_STRIDE 0x80 +#define VINTF_PAGE_SIZE 0x10000 + typedef struct Tegra241CMDQV { struct iommu_viommu_tegra241_cmdqv cmdqv_data; SMMUv3AccelState *s_accel; MemoryRegion mmio_cmdqv; qemu_irq irq; IOMMUFDVeventq *veventq; + void *vintf_page0; /* CMDQ-V Config page register cache */ uint32_t config;
hw/arm/tegra241-cmdqv.c +47 -3
88 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer,
Quoted 3 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the
s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -
Quoted 5 lines
VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.
The kernel provides an mmap offset for the VINTF Page0 region during
for the host VINTF page0 allocated for the VM
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...
Furthermore, what is not straightforward in this description is the sequence that will be executed
- viommu_alloc done on which trap event?
- hw_queue_alloc on which trap event?
- mmap on guest VINT enable
Quoted 4 lines
vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after
HW_QUEUEallocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.
Quoted 8 lines
When VINTF is enabled, mmap the kernel provided Page0 region and set
ENABLE_OKonly if the mmap succeeds. Unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation support.Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Thanks
Eric
Quoted 123 lines
--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 50 insertions(+), 3 deletions(-)hw/arm/tegra241-cmdqv.h +3
-0@@ -37,12 +37,15 @@#define CMDQV_VINTF_PAGE1_BASE 0x40000#define CMDQV_VCMDQ_STRIDE 0x80 +#define VINTF_PAGE_SIZE 0x10000 + typedef struct Tegra241CMDQV { struct iommu_viommu_tegra241_cmdqv cmdqv_data; SMMUv3AccelState *s_accel; MemoryRegion mmio_cmdqv; qemu_irq irq; IOMMUFDVeventq *veventq; + void *vintf_page0; /* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c +47
-3@@ -9,6 +9,8 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "qemu/error-report.h" +#include "qapi/error.h" #include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" @@ -228,8 +230,42 @@ static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,trace_tegra241_cmdqv_write_vcmdq_page1 (index, offset0, value); } +static bool +tegra241_cmdqv_munmap_vintf_page0 (Tegra241CMDQV *cmdqv,Error **errp) +{ +if (!cmdqv->vintf_page0) { +return true; + } + +if (munmap (cmdqv->vintf_page0,VINTF_PAGE_SIZE ) <0 ) { +error_setg_errno (errp, errno,"Failed to unmap VINTF page0" ); +return false; + } + cmdqv->vintf_page0 =NULL ; +return true; +} + +static bool tegra241_cmdqv_mmap_vintf_page0 (Tegra241CMDQV *cmdqv,Error **errp) +{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; + +if (cmdqv->vintf_page0) { +return true; + } + +if (!iommufd_backend_viommu_mmap (viommu->iommufd, viommu->viommu_id, +VINTF_PAGE_SIZE , + cmdqv->cmdqv_data.out_vintf_mmap_offset, + &cmdqv->vintf_page0, errp)) { +return false; + } + +return true; +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv, -hwaddr offset,uint64_t value) +hwaddr offset,uint64_t value, +Error **errp) {int i;@@ -244,8 +280,11 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, cmdqv->vintf_config = value;if (value &R_VINTF0_CONFIG_ENABLE_MASK ) { - cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { + cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; + } }else { +tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ; }break ;@@ -347,6 +386,7 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +Error *local_err =NULL ;int index;switch (offset) {@@ -365,7 +405,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, cmdqv->cmdq_alloc_map[(offset -A_CMDQ_ALLOC_MAP_0 ) /4 ] = value;break ; case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3: -tegra241_cmdqv_config_vintf_write (cmdqv, offset, value); +tegra241_cmdqv_config_vintf_write (cmdqv, offset, value, &local_err);break ; case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:/* @@ -398,6 +438,10 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset); } + +if (local_err) { +error_report_err (local_err); + } }/*
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Eric,
Quoted 27 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 3 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the
s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -
Quoted 4 lines
VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.
Quoted 8 lines
Quoted 1 line
The kernel provides an mmap offset for the VINTF Page0 region during
for the host VINTF page0 allocated for the VM
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:
/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/
I will populate the details and share it for review before v6.
Quoted 4 lines
when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...
Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:
- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.
- Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV hardware checks it on every command.
- The kernel allocates the VINTF with HYP_OWN = 0, which restricts the guest to a safe subset of commands.
- Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.
- IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so a guest cannot reach a VCMDQ that belongs to another VM.
Quoted 8 lines
Furthermore, what is not straightforward in this description is the sequence that will be executed
-viommu_allocdone on which trap event?
-hw_queue_allocon which trap event?
- mmap on guest VINT enableQuoted 3 lines
vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after
HW_QUEUEallocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.
Ok. Will capture the sequence of operations and the surrounding behaviour in the same top of file comment in hw/arm/tegra241-cmdqv.c.
Thanks,
Shameer
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer,
Quoted 54 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 27 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 3 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the
s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -
Quoted 4 lines
VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.
Quoted 8 lines
Quoted 1 line
The kernel provides an mmap offset for the VINTF Page0 region during
for the host VINTF page0 allocated for the VM
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of
hw/arm/tegra241-cmdqv.c:/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/I will populate the details and share it for review before v6.
Quoted 3 lines
when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...
Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:
Quoted 6 lines
- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.
- Each FD gets its own VINTF, allocated by
IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV
s/writes/configures
Quoted 11 lines
hardware checks it on every command.
- The kernel allocates the VINTF with
HYP_OWN= 0, which restricts the guest to a safe subset of commands.- Per VINTF, the kernel programs
SID_MATCHandSID_REPLACEso only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.-
IOMMU_HW_QUEUE_ALLOCbinds each VCMDQ to one specific VINTF, so a guest cannot reach a VCMDQ that belongs to another VM.
at the moment a single VCMDQ is used by VM, is that a correct understanding?
Yes this general principle explanation sound very useful to me to help the reader understand the passthrough principle.
Quoted 6 lines
Quoted 6 lines
Furthermore, what is not straightforward in this description is the sequence that will be executed
-viommu_allocdone on which trap event?
-hw_queue_allocon which trap event?
- mmap on guest VINT enable
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after
HW_QUEUEallocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.Ok. Will capture the sequence of operations and the surrounding behaviour in the same top of file comment in
hw/arm/tegra241-cmdqv.c.Thanks,
Shameer
Thanks
Eric
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Quoted 93 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 09:00
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 13 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTFPage0
Quoted 42 lines
Quoted 17 lines
for CMDQV
External email: Use caution opening links or attachments
Hi Shameer,
Quoted 4 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the
s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -
Quoted 4 lines
VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.
Quoted 8 lines
Quoted 1 line
The kernel provides an mmap offset for the VINTF Page0 region during
for the host VINTF page0 allocated for the VM
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of
hw/arm/tegra241-cmdqv.c:/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/I will populate the details and share it for review before v6.
Quoted 3 lines
when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...
Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:
Quoted 6 lines
- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.
- Each FD gets its own VINTF, allocated by
IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQVs/writes/configures
Quoted 9 lines
hardware checks it on every command.
- The kernel allocates the VINTF with
HYP_OWN= 0, which restricts the guest to a safe subset of commands.- Per VINTF, the kernel programs
SID_MATCHandSID_REPLACEso only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.
Quoted 5 lines
Quoted 1 line
-
IOMMU_HW_QUEUE_ALLOCbinds each VCMDQ to one specific VINTF, soa
Quoted 1 line
guest cannot reach a VCMDQ that belongs to another VM.
at the moment a single VCMDQ is used by VM, is that a correct understanding?
Not quite. QEMU exposes one VINTF (VINTF0) with two VCMDQs to the guest. The guest decides how many of them to enable. A Linux guest enables both:
...
arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq0 arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq1
...
Thanks,
Shameer
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Quoted 95 lines · Shameer Kolothum Thodi
Quoted 95 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 09:00
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 13 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTFPage0
Quoted 42 lines
Quoted 17 lines
for CMDQV
External email: Use caution opening links or attachments
Hi Shameer,
Quoted 4 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the
s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -
Quoted 4 lines
VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.
Quoted 8 lines
Quoted 1 line
The kernel provides an mmap offset for the VINTF Page0 region during
for the host VINTF page0 allocated for the VM
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of
hw/arm/tegra241-cmdqv.c:/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/I will populate the details and share it for review before v6.
Quoted 3 lines
when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...
Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:
Quoted 6 lines
- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.
- Each FD gets its own VINTF, allocated by
IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQVs/writes/configures
Quoted 10 lines
hardware checks it on every command.
- The kernel allocates the VINTF with
HYP_OWN= 0, which restricts the guest to a safe subset of commands.- Per VINTF, the kernel programs
SID_MATCHandSID_REPLACEso only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.-
IOMMU_HW_QUEUE_ALLOCbinds each VCMDQ to one specific VINTF, soa
Quoted 1 line
guest cannot reach a VCMDQ that belongs to another VM.
Quoted 8 lines
Quoted 2 lines
at the moment a single VCMDQ is used by VM, is that a correct understanding?
Not quite. QEMU exposes one VINTF (VINTF0) with two VCMDQs to the guest. The guest decides how many of them to enable. A Linux guest enables both:
...
arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq0 arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq1
understood!
Eric
Quoted 4 lines
...
Thanks,
Shameer
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
In reply to Shameer Kolothum ThodiQuoted 10 lines
Quoted 6 lines · Shameer Kolothum
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of
hw/arm/tegra241-cmdqv.c:
Quoted 7 lines
/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/I will populate the details and share it for review before v6.
Please find below. Hopefully, I have captured all the important details. Please take a look and let me know.
Thanks,
Shameer
hw/arm/tegra241-cmdqv.c +78 -0
@@ -7,6 +7,84 @@* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Tegra241 CMDQV - overview +* ========================= +* +* NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V) +* block. It lets a guest issue SMMU invalidation commands directly to +* dedicated hardware queues (vCMDQs) without trapping into the hypervisor on +* the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the +* host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd. +* QEMU emulates the CMDQV MMIO region and drives the host kernel calls +* (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing happens +* on real hardware. +* +* MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) +* ----------------------------------------------------- +* The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF +* apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots +* under different addressing. +* +* 0x00000 CMDQV Config page: QEMU-trapped. +* 0x10000 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed +* via vintf_ptr() to either the mmap'd VINTF page (allocated +* slot) or a per-vCMDQ register cache (unallocated slot). +* 0x20000 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped. +* 0x30000 VINTF Page 0 (per-VINTF control/status): mmap'd from the host +* via iommufd and installed into guest MMIO as a RAM-device +* subregion after the first HW_QUEUE_ALLOC; subsequent accesses +* bypass QEMU. +* 0x40000 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. +* +* The direct vCMDQ aperture stays trapped (rather than aliased to the VINTF +* mmap) to preserve the spec's R/W register semantics for unallocated +* vCMDQs: the direct aperture allows programming before VINTF allocation, +* while aliasing would route through the VINTF drop path instead. +* +* Lifecycle (driven by guest events) +* ---------------------------------- +* 1. First vfio-pci device attach (.set_iommu_device) triggers: +* - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support. +* - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, +* configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, +* forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0. +* +* 2. Guest writes VINTF_CONFIG.ENABLE = 1: +* QEMU mmap()s the offset from step 1 into its address space and reports +* STATUS.ENABLE_OK = 1. The host VINTF was already enabled by +* IOMMU_VIOMMU_ALLOC; QEMU only acks back. +* +* 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN, +* VINTF.ENABLE, in any order; each precondition write retries the +* allocation): +* IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through +* stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF. +* +* 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is +* installed into guest MMIO as a RAM-device subregion. Guest vCMDQ Page 0 +* accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter +* go straight to host hardware, bypassing QEMU. +* +* Per-VM isolation +* ---------------- +* - Each VM has its own iommufd FD; all iommufd objects (VINTF, hw_queues, +* mmap regions) belong to that FD. Cross-FD lookups fail, so one VM +* cannot reach another VM's IDs. +* - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV +* hardware substitutes / checks VMID on every command the guest issues. +* - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the +* guest to a safe subset of commands. +* - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE to restrict +* invalidations to the StreamIDs assigned to this VM. +* - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest +* cannot reach a vCMDQ that belongs to another VM. +* +* Limits exposed to the guest +* --------------------------- +* One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. +*/ +#include "qemu/osdep.h" #include "qemu/log.h" #include "qemu/error-report.h"
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer,
Quoted 51 lines · Shameer Kolothum Thodi
Quoted 9 lines
Quoted 6 lines · Shameer Kolothum
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of
hw/arm/tegra241-cmdqv.c:Quoted 7 lines
/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/I will populate the details and share it for review before v6.
Please find below. Hopefully, I have captured all the important details. Please take a look and let me know.
Thanks,
Shameerhw/arm/tegra241-cmdqv.c +21
-0@@ -7,6 +7,84 @@* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Tegra241 CMDQV - overview +* ========================= +* +* NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V) +* block. It lets a guest issue SMMU invalidation commands directly to +* dedicated hardware queues (vCMDQs) without trapping into the hypervisor on +* the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the +* host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd. +* QEMU emulates the CMDQV MMIO region and drives the host kernel calls +* (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing happens +* on real hardware. +* +* MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) +* ----------------------------------------------------- +* The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF +* apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots +* under different addressing. +* +* 0x00000 CMDQV Config page: QEMU-trapped. +* 0x10000 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed
hw/arm/tegra241-cmdqv.c line 40
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+* via vintf_ptr() to either the mmap'd VINTF page (allocated +* slot) or a per-vCMDQ register cache (unallocated slot). +* 0x20000 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped. +* 0x30000 VINTF Page 0 (per-VINTF control/status): mmap'd from the host +* via iommufd and installed into guest MMIO as a RAM-device +* subregion after the first HW_QUEUE_ALLOC; subsequent accesses +* bypass QEMU. +* 0x40000 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. +* +* The direct vCMDQ aperture stays trapped (rather than aliased to the VINTF
direct vCMDQ aperture page 0 stays trapped as opposed to the VIINTF page0
hw/arm/tegra241-cmdqv.c line 43
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+* mmap) to preserve the spec's R/W register semantics for unallocated +* vCMDQs: the direct aperture allows programming before VINTF allocation, +* while aliasing would route through the VINTF drop path instead.
see last discussion
hw/arm/tegra241-cmdqv.c line 51
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+* +* Lifecycle (driven by guest events) +* ---------------------------------- +* 1. First vfio-pci device attach (.set_iommu_device) triggers: +* - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support. +* - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, +* configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, +* forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
what about the v/p SID mapping. How does the kernel know which SIDs are supposed to write into that VINTF? where do we pass this info?
Quoted 29 lines
hw/arm/tegra241-cmdqv.c +29
-0+* +* 2. Guest writes VINTF_CONFIG.ENABLE = 1: +* QEMU mmap()s the offset from step 1 into its address space and reports +* STATUS.ENABLE_OK = 1. The host VINTF was already enabled by +* IOMMU_VIOMMU_ALLOC; QEMU only acks back. +* +* 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN, +* VINTF.ENABLE, in any order; each precondition write retries the +* allocation): +* IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through +* stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF. +* +* 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is +* installed into guest MMIO as a RAM-device subregion. Guest vCMDQ Page 0 +* accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter +* go straight to host hardware, bypassing QEMU. +* +* Per-VM isolation +* ---------------- +* - Each VM has its own iommufd FD; all iommufd objects (VINTF, hw_queues, +* mmap regions) belong to that FD. Cross-FD lookups fail, so one VM +* cannot reach another VM's IDs. +* - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV +* hardware substitutes / checks VMID on every command the guest issues. +* - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the +* guest to a safe subset of commands. +* - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE to restrict +* invalidations to the StreamIDs assigned to this VM. +* - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest
hw/arm/tegra241-cmdqv.c line 90
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +7
-0+* cannot reach a vCMDQ that belongs to another VM. +* +* Limits exposed to the guest +* --------------------------- +* One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. +*/ +#include "qemu/osdep.h" #include "qemu/log.h" #include "qemu/error-report.h"
Eric
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Quoted 92 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:32
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 27 lines · Shameer Kolothum Thodi
Quoted 9 lines
Quoted 6 lines · Shameer Kolothum
Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.
Fair point. A summary of operation is useful. I am thinking of adding it at the top of
hw/arm/tegra241-cmdqv.c:Quoted 7 lines
/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/I will populate the details and share it for review before v6.
Please find below. Hopefully, I have captured all the important details. Please take a look and let me know.
Thanks,
Shameerhw/arm/tegra241-cmdqv.c +0
-0hw/arm/tegra241-cmdqv.c line 14
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +5
-0@@ -7,6 +7,84 @@* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Tegra241 CMDQV - overview +* ========================= +* +* NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V)
hw/arm/tegra241-cmdqv.c line 16
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+* block. It lets a guest issue SMMU invalidation commands directly to +* dedicated hardware queues (vCMDQs) without trapping into the hypervisor on
hw/arm/tegra241-cmdqv.c line 18
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+* the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the +* host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd.
hw/arm/tegra241-cmdqv.c line 19
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+* QEMU emulates the CMDQV MMIO region and drives the host kernel calls
hw/arm/tegra241-cmdqv.c line 20
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+* (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing happens
hw/arm/tegra241-cmdqv.c
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+* on real hardware. +* +* MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) +* ----------------------------------------------------- +* The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF
hw/arm/tegra241-cmdqv.c line 26
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+* apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots
hw/arm/tegra241-cmdqv.c
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+* under different addressing. +* +* 0x00000 CMDQV Config page: QEMU-trapped. +* 0x10000 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed
hw/arm/tegra241-cmdqv.c
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+* via vintf_ptr() to either the mmap'd VINTF page (allocated +* slot) or a per-vCMDQ register cache (unallocated slot). +* 0x20000 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU- trapped.
hw/arm/tegra241-cmdqv.c line 34
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+* 0x30000 VINTF Page 0 (per-VINTF control/status): mmap'd from the host
hw/arm/tegra241-cmdqv.c
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+* via iommufd and installed into guest MMIO as a RAM-device +* subregion after the first HW_QUEUE_ALLOC; subsequent accesses +* bypass QEMU. +* 0x40000 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. +* +* The direct vCMDQ aperture stays trapped (rather than aliased to the
Quoted 3 lines
VINTF
direct vCMDQ aperture page 0 stays trapped as opposed to the VIINTF page0
Ok.
Quoted 6 lines
hw/arm/tegra241-cmdqv.c line 42
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +2
-0+* mmap) to preserve the spec's R/W register semantics for unallocated +* vCMDQs: the direct aperture allows programming before VINTF allocation,
hw/arm/tegra241-cmdqv.c line 43
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+* while aliasing would route through the VINTF drop path instead. see last discussion
Replied there.
Quoted 5 lines
hw/arm/tegra241-cmdqv.c
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+* +* Lifecycle (driven by guest events) +* ---------------------------------- +* 1. First vfio-pci device attach (.set_iommu_device) triggers: +* - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host
Quoted 7 lines
CMDQV support.
hw/arm/tegra241-cmdqv.c
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+* - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, +* configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, +* forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
what about the v/p SID mapping. How does the kernel know which SIDs are supposed to write into that VINTF? where do we pass this info?
That happens later, driven by the guest SMMU driver, not at vfio-pci device attach.
Flow:
- Guest SMMU driver configures the Stream Table Entry (STE) for the device and issues CMD_CFGI_STE on the SMMU command queue.
- QEMU calls smmuv3_accel_install_ste(), which in turn calls iommufd_backend_alloc_vdev() -> IOMMU_VDEVICE_ALLOC ioctl.
- iommufd core (iommufd_vdevice_alloc_ioctl) dispatches to the Tegra CMDQV driver via viommu->ops->vdevice_init, which is tegra241_vintf_init_vsid().
- That writes the host stream id into SID_REPLACE and the guest virtual SID into SID_MATCH.
I will summarise and include this in the overview.
Thanks,
Shameer
PATCH v5 18/32 system/physmem: Add address_space_range_is_ram() helper
In reply to Shameer KolothumIntroduce address_space_range_is_ram(), a helper to determine whether a guest physical address range within an AddressSpace resolves entirely to RAM-backed MemoryRegions. The range is walked using address_space_translate() so contiguous RAM-only ranges return true even when they cross multiple MemoryRegion boundaries; any byte backed by non-RAM (MMIO, ROM device, hole) returns false.
--- include/system/memory.h | 13 +++++++++++++ system/physmem.c | 19 +++++++++++++++++++ 2 files changed, 32 insertions(+)
include/system/memory.h +13 -0
@@ -2841,6 +2841,19 @@ bool address_space_access_valid(AddressSpace *as, hwaddr addr, hwaddr len,*/ bool address_space_is_io (AddressSpace *as,hwaddr addr); +/** +* address_space_range_is_ram: check whether a guest physical address range +* within an address space resolves entirely to RAM. +* +* @as: #AddressSpace to be accessed +* @addr: address within that address space +* @size: number of bytes in the range starting at @addr +* +* Returns true if every byte in [@addr, @addr + @size) maps to a RAM +* MemoryRegion, false otherwise. +*/ +bool address_space_range_is_ram (AddressSpace *as,hwaddr addr,hwaddr size); +/* address_space_map: map a physical memory region into a host virtual address * * May map a subset of the requested range, given by and returned in @plen.
system/physmem.c +19 -0
@@ -3668,6 +3668,25 @@ bool address_space_is_io(AddressSpace *as, hwaddr addr)return !(memory_region_is_ram (mr) ||memory_region_is_romd (mr)); } +bool address_space_range_is_ram (AddressSpace *as,hwaddr addr,hwaddr size) +{ +MemoryRegion *mr; +hwaddr xlat, l; + +RCU_READ_LOCK_GUARD (); +while (size >0 ) { + l = size; + mr =address_space_translate (as, addr, &xlat, &l, false, +MEMTXATTRS_UNSPECIFIED ); +if (!memory_region_is_ram (mr)) { +return false; + } + size -= l; + addr += l; + } +return true; +} + static hwaddr flatview_extend_translation(FlatView *fv, hwaddr addr, hwaddr target_len,
Signature · 1 line
2.43.0
PATCH v5 19/32 hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.
Writes with invalid addresses are ignored.
All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
--- hw/arm/tegra241-cmdqv.h | 11 ++++ hw/arm/tegra241-cmdqv.c | 138 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 143 insertions(+), 6 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */
@@ -361,6 +362,16 @@ SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +132 -6
220 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Hi,
Quoted 4 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
You should add the fact that
iommufd_backend_alloc_hw_queue() requires the GPA of the VCMDQ. This explains why you need to those operations once the GPA has been populated.
Quoted 106 lines
VCMDQ_ENlives inVCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.Writes with invalid addresses are ignored.
All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 11 ++++ hw/arm/tegra241-cmdqv.c | 138 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 143 insertions(+), 6 deletions(-)hw/arm/tegra241-cmdqv.h +11
-0@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */ @@ -361,6 +362,16 @@ SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +46
-0@@ -18,6 +18,95 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) +{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; +IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index]; + +if (!vcmdq) { +return ; + } +iommufd_backend_free_id (viommu->iommufd, vcmdq->hw_queue_id); +g_free (vcmdq); + cmdqv->vcmdq[index] =NULL ; +} + +static void tegra241_cmdqv_free_all_vcmdq (Tegra241CMDQV *cmdqv) +{ +/* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */ +for (int i = (TEGRA241_CMDQV_MAX_CMDQ -1 ); i >=0 ; i--) { +tegra241_cmdqv_free_vcmdq (cmdqv, i); + } +} + +/* +* Allocate a host HW VCMDQ from the current cached BASE / size for @index. +* +* Setup is a no-op (returns true) when any of the following preconditions +* isn't met yet: +* - BASE not prgrammed yet. +* - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0) +* - BASE / size don't resolve to a RAM region +* - CMDQV global enable or VINTF enable is not yet asserted +*/ +static bool tegra241_cmdqv_setup_vcmdq (Tegra241CMDQV *cmdqv,int index, +Error **errp) +{ +SMMUv3AccelState *accel = cmdqv->s_accel; +uint64_t base_mask = (uint64_t )R_VCMDQ0_BASE_L_ADDR_MASK | + (uint64_t )R_VCMDQ0_BASE_H_ADDR_MASK <<32 ; +uint64_t addr = cmdqv->vcmdq_base[index] & base_mask; +uint64_t log2 = cmdqv->vcmdq_base[index] &R_VCMDQ0_BASE_L_LOG2SIZE_MASK ; +uint64_t size =1ULL << (log2 +4 ); +IOMMUFDViommu *viommu = accel->viommu; +IOMMUFDHWqueue *hw_queue; +uint32_t hw_queue_id; + +/* BASE not programmed yet. */ +if (!cmdqv->vcmdq_base[index]) {
hw/arm/tegra241-cmdqv.c line 76
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+return true; + } + +/* VCMDQ not yet mapped to a VINTF. */ +if (!(cmdqv->cmdq_alloc_map[index] &R_CMDQ_ALLOC_MAP_0_ALLOC_MASK )) { +return true; + } + +/* Ignore any invalid address. This may come as part of reset etc. */ +if (!address_space_range_is_ram (&address_space_memory, addr, size)) {
why do you need to test the whole range and not only the start address
Quoted 18 lines
hw/arm/tegra241-cmdqv.c +18
-0+return true; + } + +if (!tegra241_cmdqv_enabled (cmdqv) || !tegra241_vintf_enabled (cmdqv)) { +return true; + } + +tegra241_cmdqv_free_vcmdq (cmdqv, index); + +if (!iommufd_backend_alloc_hw_queue (viommu->iommufd, viommu->viommu_id, +IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV , + index, addr, size, &hw_queue_id, + errp)) { +return false; + } + hw_queue =g_new (IOMMUFDHWqueue,1 ); + hw_queue->hw_queue_id = hw_queue_id; + hw_queue->viommu = viommu;
hw/arm/tegra241-cmdqv.c line 104
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ cmdqv->vcmdq[index] = hw_queue; + +return true; +} + +static void tegra241_cmdqv_setup_all_vcmdq (Tegra241CMDQV *cmdqv, +Error **errp) +{ +for (int i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { +if (!tegra241_cmdqv_setup_vcmdq (cmdqv, i, errp)) {
if case of failure,what is the functional consequence. Can you comment on this in the commit msg or here.
Quoted 133 lines
hw/arm/tegra241-cmdqv.c +44
-6+return ; + } + } +} +/* * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. */ @@ -172,7 +261,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,*/ static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index, -uint32_t value) +uint32_t value,Error **errp) {switch (offset0) {case A_VCMDQ0_BASE_L :@@ -181,6 +270,7 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv, } cmdqv->vcmdq_base[index] =deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +tegra241_cmdqv_setup_vcmdq (cmdqv, index, errp);break ;case A_VCMDQ0_BASE_H :if (tegra241_vcmdq_enabled (cmdqv, index)) {@@ -188,6 +278,7 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv, } cmdqv->vcmdq_base[index] =deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +tegra241_cmdqv_setup_vcmdq (cmdqv, index, errp);break ;case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : cmdqv->vcmdq_cons_indx_base[index] =@@ -210,7 +301,7 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,*/ static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index, -uint64_t value) +uint64_t value,Error **errp) {switch (offset0) {case A_VCMDQ0_BASE_L :@@ -218,6 +309,7 @@ static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,return ; } cmdqv->vcmdq_base[index] = value; +tegra241_cmdqv_setup_vcmdq (cmdqv, index, errp);break ;case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : cmdqv->vcmdq_cons_indx_base[index] = value;@@ -282,8 +374,14 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,if (value &R_VINTF0_CONFIG_ENABLE_MASK ) {if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +/* +* VCMDQs whose BASE was programmed before VINTF was +* enabled need their hw_queue allocated now. +*/ +tegra241_cmdqv_setup_all_vcmdq (cmdqv, errp); } }else { +tegra241_cmdqv_free_all_vcmdq (cmdqv);tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ; }@@ -394,16 +492,37 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, cmdqv->config = value;if (value &R_CONFIG_CMDQV_EN_MASK ) { cmdqv->status |=R_STATUS_CMDQV_ENABLED_MASK ; +/* +* VCMDQs whose BASE was programmed before CMDQV was enabled +* need their hw_queue allocated now. +*/ +tegra241_cmdqv_setup_all_vcmdq (cmdqv, &local_err); }else { +tegra241_cmdqv_free_all_vcmdq (cmdqv); cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK ; }break ; case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1: cmdqv->vi_int_mask[(offset -A_VI_INT_MASK_0 ) /4 ] = value;break ; - case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: - cmdqv->cmdq_alloc_map[(offset -A_CMDQ_ALLOC_MAP_0 ) /4 ] = value; + case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: { +int idx = (offset -A_CMDQ_ALLOC_MAP_0 ) /4 ; +bool was_alloc = cmdqv->cmdq_alloc_map[idx] & +R_CMDQ_ALLOC_MAP_0_ALLOC_MASK ; +bool now_alloc = value &R_CMDQ_ALLOC_MAP_0_ALLOC_MASK ; + + cmdqv->cmdq_alloc_map[idx] = value; +/* +* If the VCMDQ was already programmed (BASE) before mapping, fire +* setup on the ALLOC 0->1 transition; tear down on 1->0. +*/ +if (!was_alloc && now_alloc) { +tegra241_cmdqv_setup_vcmdq (cmdqv, idx, &local_err); + }else if (was_alloc && !now_alloc) { +tegra241_cmdqv_free_vcmdq (cmdqv, idx); + }break ; + } case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value, &local_err);break ;@@ -432,7 +551,8 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ;tegra241_cmdqv_write_vcmdq_page1 (cmdqv, - offset - index *CMDQV_VCMDQ_STRIDE , index, value); + offset - index *CMDQV_VCMDQ_STRIDE , index, value, + &local_err);break ;default : qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",@@ -451,6 +571,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { +Error *local_err =NULL ;int index;switch (offset) {@@ -465,13 +586,18 @@ static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ;tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, - offset - index *CMDQV_VCMDQ_STRIDE , index, value); + offset - index *CMDQV_VCMDQ_STRIDE , index, value, + &local_err);break ;default : qemu_log_mask(LOG_UNIMP,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 603
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +4
-0"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); } + +if (local_err) { +error_report_err (local_err); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
Thanks
Eric
RE: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 21 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 10:30
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configuredExternal email: Use caution opening links or attachments
Hi,
On 5/19/26 12:36 PM, Shameer Kolothum wrote:
Quoted 8 lines
Quoted 4 lines
From: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
You should add the fact that
iommufd_backend_alloc_hw_queue()requires the GPA of the VCMDQ. This explains why you need to those operations once the GPA has been populated.
Ok.
Quoted 120 lines
Quoted 5 lines
VCMDQ_ENlives inVCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV /VINTF
Quoted 10 lines
enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.Writes with invalid addresses are ignored.
All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 11 ++++ hw/arm/tegra241-cmdqv.c | 138++++++++++++++++++++++++++++++++++++++--
hw/arm/tegra241-cmdqv.h
Quoted 15 lines
2 files changed, 143 insertions(+), 6 deletions(-)hw/arm/tegra241-cmdqv.h +1
-0@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */ @@ -361,6 +362,16 @@
SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)Quoted 21 lines
hw/arm/tegra241-cmdqv.h +10
-0SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +0
-0hw/arm/tegra241-cmdqv.c line 21
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -18,6 +18,95 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, intindex)
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; +IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index]; + +if (!vcmdq) {hw/arm/tegra241-cmdqv.c
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+return ; + } +iommufd_backend_free_id (viommu->iommufd, vcmdq->hw_queue_id); +g_free (vcmdq); + cmdqv->vcmdq[index] =NULL ; +} + +static void tegra241_cmdqv_free_all_vcmdq (Tegra241CMDQV *cmdqv) +{ +/* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */
hw/arm/tegra241-cmdqv.c
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7
-0+for (int i = (TEGRA241_CMDQV_MAX_CMDQ -1 ); i >=0 ; i--) { +tegra241_cmdqv_free_vcmdq (cmdqv, i); + } +} + +/* +* Allocate a host HW VCMDQ from the current cached BASE / size for @index.
hw/arm/tegra241-cmdqv.c
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9
-0+* +* Setup is a no-op (returns true) when any of the following preconditions +* isn't met yet: +* - BASE not prgrammed yet. +* - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0) +* - BASE / size don't resolve to a RAM region +* - CMDQV global enable or VINTF enable is not yet asserted +*/ +static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, intindex,
hw/arm/tegra241-cmdqv.c
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7
-0+ Error **errp) +{ +SMMUv3AccelState *accel = cmdqv->s_accel; +uint64_t base_mask = (uint64_t )R_VCMDQ0_BASE_L_ADDR_MASK | + (uint64_t )R_VCMDQ0_BASE_H_ADDR_MASK <<32 ; +uint64_t addr = cmdqv->vcmdq_base[index] & base_mask; + uint64_t log2 = cmdqv->vcmdq_base[index] &
R_VCMDQ0_BASE_L_LOG2SIZE_MASK;hw/arm/tegra241-cmdqv.c
Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12
-0+uint64_t size =1ULL << (log2 +4 ); +IOMMUFDViommu *viommu = accel->viommu; +IOMMUFDHWqueue *hw_queue; +uint32_t hw_queue_id; + +/* BASE not programmed yet. */ +if (!cmdqv->vcmdq_base[index]) { +return true; + } + +/* VCMDQ not yet mapped to a VINTF. */ + if (!(cmdqv->cmdq_alloc_map[index] &
R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {
Quoted 6 lines
hw/arm/tegra241-cmdqv.c line 76
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+return true; + } + +/* Ignore any invalid address. This may come as part of reset etc. */ +if (!address_space_range_is_ram (&address_space_memory, addr, size)) {why do you need to test the whole range and not only the start address
That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/
Please let me know if there is a better way.
Quoted 24 lines
hw/arm/tegra241-cmdqv.c
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+return true; + } + + if (!tegra241_cmdqv_enabled(cmdqv) ||!tegra241_vintf_enabled(cmdqv)) {
Quoted 19 lines
hw/arm/tegra241-cmdqv.c +18
-0+return true; + } + +tegra241_cmdqv_free_vcmdq (cmdqv, index); + +if (!iommufd_backend_alloc_hw_queue (viommu->iommufd, viommu- viommu_id, +IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV , + index, addr, size, &hw_queue_id, + errp)) { +return false; + } + hw_queue =g_new (IOMMUFDHWqueue,1 ); + hw_queue->hw_queue_id = hw_queue_id; + hw_queue->viommu = viommu; + cmdqv->vcmdq[index] = hw_queue; + +return true; +}
Quoted 8 lines
hw/arm/tegra241-cmdqv.c line 104
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+ +static void tegra241_cmdqv_setup_all_vcmdq (Tegra241CMDQV *cmdqv, +Error **errp) +{ +for (int i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { +if (!tegra241_cmdqv_setup_vcmdq (cmdqv, i, errp)) {if case of failure,what is the functional consequence. Can you comment on this in the commit msg or here.
Today QEMU only logs the error:
qemu-system-aarch64: IOMMU_HW_QUEUE_ALLOC failed: error -22
The guest visible behaviour then depends on whether VINTF0 Page 0 was mapped into guest memory before this failure:
Case 1: setup_vcmdq() succeeds for vcmdq0, fails for vcmdq1:
- VINTF0 Page 0 was mapped to the guest after vcmdq0's success.
- Guest writes VCMDQ1_CONFIG go directly to HW.
- Guest reads VCMDQ1_STATUS directly from HW:
...
tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: failed to enable, STATUS=0x00000000
tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: GERRORN=0x0, GERROR=0x0, CONS=0x0
...Case 2: setup_vcmdq() fails for vcmdq0:
- VINTF0 Page 0 is not mapped to the guest.
- Guest writes VCMDQ0_CONFIG. Trapped by QEMU.
- Guest reads VCMDQ0_STATUS. Trapped by QEMU.
- Both served from the register cache. The cache currently doesn't reflect the failure, so the guest thinks the queue is enabled and proceeds to issue commands, eventually:
arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000002 [hwprod 0x00000002, hwcons 0x00000000]
arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000004 [hwprod 0x00000004, hwcons 0x00000000]Case 2 needs fixing. I'll fix the register cache on the error path for v6 and document it.
Thanks,
Shameer
Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 154 lines · Shameer Kolothum Thodi
Quoted 20 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 10:30
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configuredExternal email: Use caution opening links or attachments
Hi,
On 5/19/26 12:36 PM, Shameer Kolothum wrote:
Quoted 8 lines
Quoted 4 lines
From: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
You should add the fact that
iommufd_backend_alloc_hw_queue()requires the GPA of the VCMDQ. This explains why you need to those operations once the GPA has been populated.Ok.
Quoted 124 lines
Quoted 4 lines
VCMDQ_ENlives inVCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV /VINTF
Quoted 10 lines
enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.Writes with invalid addresses are ignored.
All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 11 ++++ hw/arm/tegra241-cmdqv.c | 138++++++++++++++++++++++++++++++++++++++--
hw/arm/tegra241-cmdqv.h
Quoted 15 lines
2 files changed, 143 insertions(+), 6 deletions(-)hw/arm/tegra241-cmdqv.h +1
-0@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */ @@ -361,6 +362,16 @@
SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)Quoted 21 lines
hw/arm/tegra241-cmdqv.h +10
-0SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +0
-0hw/arm/tegra241-cmdqv.c line 21
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -18,6 +18,95 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, intindex)
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; +IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index]; + +if (!vcmdq) {hw/arm/tegra241-cmdqv.c
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+return ; + } +iommufd_backend_free_id (viommu->iommufd, vcmdq->hw_queue_id); +g_free (vcmdq); + cmdqv->vcmdq[index] =NULL ; +} + +static void tegra241_cmdqv_free_all_vcmdq (Tegra241CMDQV *cmdqv) +{ +/* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */
hw/arm/tegra241-cmdqv.c
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7
-0+for (int i = (TEGRA241_CMDQV_MAX_CMDQ -1 ); i >=0 ; i--) { +tegra241_cmdqv_free_vcmdq (cmdqv, i); + } +} + +/* +* Allocate a host HW VCMDQ from the current cached BASE / size for @index.
hw/arm/tegra241-cmdqv.c
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9
-0+* +* Setup is a no-op (returns true) when any of the following preconditions +* isn't met yet: +* - BASE not prgrammed yet. +* - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0) +* - BASE / size don't resolve to a RAM region +* - CMDQV global enable or VINTF enable is not yet asserted +*/ +static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, intindex,
hw/arm/tegra241-cmdqv.c
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7
-0+ Error **errp) +{ +SMMUv3AccelState *accel = cmdqv->s_accel; +uint64_t base_mask = (uint64_t )R_VCMDQ0_BASE_L_ADDR_MASK | + (uint64_t )R_VCMDQ0_BASE_H_ADDR_MASK <<32 ; +uint64_t addr = cmdqv->vcmdq_base[index] & base_mask; + uint64_t log2 = cmdqv->vcmdq_base[index] &
R_VCMDQ0_BASE_L_LOG2SIZE_MASK;hw/arm/tegra241-cmdqv.c
Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12
-0+uint64_t size =1ULL << (log2 +4 ); +IOMMUFDViommu *viommu = accel->viommu; +IOMMUFDHWqueue *hw_queue; +uint32_t hw_queue_id; + +/* BASE not programmed yet. */ +if (!cmdqv->vcmdq_base[index]) { +return true; + } + +/* VCMDQ not yet mapped to a VINTF. */ + if (!(cmdqv->cmdq_alloc_map[index] &
R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+return true; + } + +/* Ignore any invalid address. This may come as part of reset etc. */ +if (!address_space_range_is_ram (&address_space_memory, addr, size)) {
Quoted 5 lines
Quoted 1 line
why do you need to test the whole range and not only the start address
That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/Please let me know if there is a better way.
Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin
Eric
Quoted 60 lines
Quoted 25 lines
hw/arm/tegra241-cmdqv.c
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +4
-0+return true; + } + + if (!tegra241_cmdqv_enabled(cmdqv) ||!tegra241_vintf_enabled(cmdqv)) {
Quoted 19 lines
hw/arm/tegra241-cmdqv.c +18
-0+return true; + } + +tegra241_cmdqv_free_vcmdq (cmdqv, index); + +if (!iommufd_backend_alloc_hw_queue (viommu->iommufd, viommu- viommu_id, +IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV , + index, addr, size, &hw_queue_id, + errp)) { +return false; + } + hw_queue =g_new (IOMMUFDHWqueue,1 ); + hw_queue->hw_queue_id = hw_queue_id; + hw_queue->viommu = viommu; + cmdqv->vcmdq[index] = hw_queue; + +return true; +}Quoted 8 lines
hw/arm/tegra241-cmdqv.c line 104
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+ +static void tegra241_cmdqv_setup_all_vcmdq (Tegra241CMDQV *cmdqv, +Error **errp) +{ +for (int i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { +if (!tegra241_cmdqv_setup_vcmdq (cmdqv, i, errp)) {if case of failure,what is the functional consequence. Can you comment on this in the commit msg or here.
Today QEMU only logs the error:
qemu-system-aarch64:
IOMMU_HW_QUEUE_ALLOCfailed: error -22The guest visible behaviour then depends on whether VINTF0 Page 0 was mapped into guest memory before this failure:
Case 1:
setup_vcmdq()succeeds for vcmdq0, fails for vcmdq1:- VINTF0 Page 0 was mapped to the guest after vcmdq0's success.
- Guest writesVCMDQ1_CONFIGgo directly to HW.
- Guest readsVCMDQ1_STATUSdirectly from HW:... tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: failed to enable, STATUS=0x00000000 tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: GERRORN=0x0, GERROR=0x0, CONS=0x0 ...Case 2:
setup_vcmdq()fails for vcmdq0:- VINTF0 Page 0 is not mapped to the guest.
- Guest writesVCMDQ0_CONFIG. Trapped by QEMU.
- Guest readsVCMDQ0_STATUS. Trapped by QEMU.
- Both served from the register cache. The cache currently doesn't reflect the failure, so the guest thinks the queue is enabled and proceeds to issue commands, eventually:
Quoted 5 lines
arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000002 [hwprod 0x00000002, hwcons 0x00000000] arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000004 [hwprod 0x00000004, hwcons 0x00000000]Case 2 needs fixing. I'll fix the register cache on the error path for v6 and document it.
OK ;-)
Thanks
Eric
Quoted 3 lines
Thanks,
Shameer
Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 9 lines · Eric Auger
Quoted 8 lines · Shameer Kolothum Thodi
Quoted 4 lines
hw/arm/tegra241-cmdqv.c line 76
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+/* Ignore any invalid address. This may come as part of reset etc. */ +if (!address_space_range_is_ram (&address_space_memory, addr, size)) {why do you need to test the whole range and not only the start addressa
That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/
Quoted 6 lines
Quoted 1 line
Please let me know if there is a better way.
Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin
Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's is_ram.
So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?
AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.
Nicolin
Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Hi Nicolin,
Quoted 13 lines · Nicolin Chen
Quoted 4 lines · Eric Auger
Quoted 3 lines · Shameer Kolothum Thodi
Quoted 3 lines
hw/arm/tegra241-cmdqv.c line 76
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+/* Ignore any invalid address. This may come as part of reset etc. */ +if (!address_space_range_is_ram (&address_space_memory, addr, size)) {why do you need to test the whole range and not only the start addressa
Quoted 8 lines
Quoted 4 lines
That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/Please let me know if there is a better way.
Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin
Quoted 8 lines
Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's
is_ram.So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?
AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.
would it jeopardize the host integrity or would it just break the misbehaving guest. If the latest I am not sure this is critical to check. We are not checking either with regular SMMU cmd queues
Thanks
Eric
Quoted 3 lines
Nicolin
RE: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 18 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:37
To: Nicolin Chen <[email protected]>
Cc: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configuredExternal email: Use caution opening links or attachments
Quoted 22 lines
Hi Nicolin,
Quoted 4 lines · Nicolin Chen
Quoted 3 lines · Eric Auger
Quoted 2 lines · Shameer Kolothum Thodi
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Ignore any invalid address. This may come as part of reset etc. */ + if (!address_space_range_is_ram(&address_space_memory, addr,size)) {
Quoted 14 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
why do you need to test the whole range and not only the start addressa
That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/Please let me know if there is a better way.
Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin
Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's
is_ram.So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?
Looking at our helper address_space_range_is_ram(), it isn't foolproof for the above cases. For the PCI BAR case we could add:
if (!memory_region_is_ram(mr) || memory_region_is_ram_device(mr)) {
return false;
}But I'm not sure how to tackle the reserved spaces. For firmware ROM we could add a memory_region_is_rom() check, but the others will still look like normal RAM, so they're hard to detect at this layer.
Quoted 6 lines
Quoted 3 lines
AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.
would it jeopardize the host integrity or would it just break the misbehaving guest. If the latest I am not sure this is critical to check. We are not checking either with regular SMMU cmd queues
I think it would just be the guest shooting itself in the foot. Can't think of a host security/integrity issue here. So we could drop the check if that's preferred.
Thanks,
Shameer
Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 50 lines · Shameer Kolothum Thodi
Quoted 17 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:37
To: Nicolin Chen <[email protected]>
Cc: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configuredExternal email: Use caution opening links or attachments
Quoted 22 lines
Hi Nicolin,
Quoted 4 lines · Nicolin Chen
Quoted 3 lines · Eric Auger
Quoted 2 lines · Shameer Kolothum Thodi
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Ignore any invalid address. This may come as part of reset etc. */ + if (!address_space_range_is_ram(&address_space_memory, addr,size)) {
Quoted 14 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
why do you need to test the whole range and not only the start addressa
That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/Please let me know if there is a better way.
Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin
Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's
is_ram.So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?
Looking at our helper
address_space_range_is_ram(), it isn't foolproof for the above cases. For the PCI BAR case we could add:if (!memory_region_is_ram(mr) || memory_region_is_ram_device(mr)) { return false; }But I'm not sure how to tackle the reserved spaces. For firmware ROM we could add a
memory_region_is_rom()check, but the others will still look like normal RAM, so they're hard to detect at this layer.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.
would it jeopardize the host integrity or would it just break the misbehaving guest. If the latest I am not sure this is critical to check. We are not checking either with regular SMMU cmd queues
I think it would just be the guest shooting itself in the foot. Can't think of a host
security/integrityissue here. So we could drop the check if that's preferred.
In that case I would rather simplify and remove the check. But that's my humble opinion ;-)
Eric
Quoted 3 lines
Thanks,
Shameer
PATCH v5 20/32 hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
In reply to Shameer KolothumIntroduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.
There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:
- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)
These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.
VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC) has been allocated for the VCMDQ:
Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register
cache. Hardware is not yet engaged.
Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.
The pre-to-post-alloc transition is triggered by the IOMMUFD hardware queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-alloc cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page so the guest's view survives the transition.
CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".
--- hw/arm/tegra241-cmdqv.c | 72 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +70 -2
@@ -39,6 +39,39 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) } } +/* Pointer into the mmap'd VINTF page0 slot for @index, or NULL. */ +static inline uint32_t *tegra241_cmdqv_vintf_ptr (Tegra241CMDQV *cmdqv, +int index,hwaddr offset0) +{ +if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) { +return NULL ; + } +return (uint32_t *)(cmdqv->vintf_page0 + + (index *CMDQV_VCMDQ_STRIDE ) + + (offset0 -CMDQV_VCMDQ_PAGE0_BASE )); +} + +/* Flush cached pre-alloc writes into the mmap'd VINTF page0 slot. */ +static void tegra241_cmdqv_sync_vcmdq (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t *ptr; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONS_INDX ); +if (!ptr) { +return ; + } + *ptr = cmdqv->vcmdq_cons_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_PROD_INDX ); + *ptr = cmdqv->vcmdq_prod_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONFIG ); + *ptr = cmdqv->vcmdq_config[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_GERRORN ); + *ptr = cmdqv->vcmdq_gerrorn[index]; +} +/* * Allocate a host HW VCMDQ from the current cached BASE / size for @index. *
@@ -94,6 +127,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, hw_queue->viommu = viommu; cmdqv->vcmdq[index] = hw_queue; +/* Pre-alloc cached writes survive the cache-to-hardware transition. */ +tegra241_cmdqv_sync_vcmdq (cmdqv, index); +return true; }
@@ -108,11 +144,14 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv, }/* -* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. Reads from the +* mmap'd VINTF page0 when the VCMDQ is allocated (HW is the source of +* truth post-alloc); otherwise reads from the register cache. */ static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) { -uint32_t status = cmdqv->vcmdq_status[index]; +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_STATUS ); +uint32_t status = ptr ? *ptr : cmdqv->vcmdq_status[index];return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; }
@@ -122,12 +161,21 @@ static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)* * The caller normalizes the MMIO offset such that @offset0 always refers * to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly +* from the VINTF page0 backing. Otherwise, fall back to the cache. */ static uint64_t tegra241_cmdqv_read_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0);uint64_t val =0 ; +if (ptr) { + val = *ptr; + goto out; + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : val = cmdqv->vcmdq_cons_indx[index];
@@ -152,6 +200,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_read_vcmdq_page0 (index, offset0, val);return val; }
@@ -225,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,* * Page 0 registers are all 32-bit; this helper is only called for 4-byte * writes. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly +* to the VINTF page0 backing. Otherwise, update the cache. */ static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index,uint32_t value) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0); + +if (ptr) { +switch (offset0) { +case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : cmdqv->vcmdq_cons_indx[index] = value;
@@ -253,6 +320,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); }
Signature · 1 line
2.43.0
Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Hi Shameer,
Quoted 13 lines · Shameer Kolothum
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:
- Direct VCMDQ Page0 aperture (offset
0x10000)
- VINTF Page0 (offset0x30000)These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.
Quoted 5 lines
VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (
IOMMU_HW_QUEUE_ALLOC) has been allocated for the VCMDQ:Pre-alloc:
vintf_ptris NULL. Both apertures use QEMU's register
do you mean vintf_page0 instead of vintf_ptr.
vintf_page0 is set on mmap, right? and not on hwqueue_alloc
Besides pre/post-alloc sounds a bit confusing to me since it can mean queue_alloc or mmap.
If I understand correctly vintf_page0 can be set while hwqueue_alloc has not been done yet and conversely, hence the double check below:
+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
Quoted 8 lines
cache. Hardware is not yet engaged.
Post-alloc:
vintf_ptris valid. Both QEMU trapped apertures access
registers directly via the mmap'dvintf_page0pointer, bypassing the cache. Hardware is the single source of truth.The pre-to-post-alloc transition is triggered by the IOMMUFD hardware
so here you mean alloc is hw queue alloc
Quoted 10 lines
queue allocation. The
tegra241_cmdqv_sync_vcmdq()copies any pre-alloc cached writes (CONS_INDX,PROD_INDX, CONFIG, GERRORN) into the mmap'd page so the guest's view survives the transition.CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".
I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32
Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?
Quoted 11 lines
Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 72 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 2 deletions(-)hw/arm/tegra241-cmdqv.c +0
-0
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) · line 42
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -39,6 +39,39 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) } } +/* Pointer into the mmap'd VINTF page0 slot for @index, or NULL. */
@index: index of the vcmdq
@offset0, offset within vcmdq page0
Quoted 123 lines
hw/arm/tegra241-cmdqv.c +68
-2+static inline uint32_t *tegra241_cmdqv_vintf_ptr (Tegra241CMDQV *cmdqv, +int index,hwaddr offset0) +{ +if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) { +return NULL ; + } +return (uint32_t *)(cmdqv->vintf_page0 + + (index *CMDQV_VCMDQ_STRIDE ) + + (offset0 -CMDQV_VCMDQ_PAGE0_BASE )); +} + +/* Flush cached pre-alloc writes into the mmap'd VINTF page0 slot. */ +static void tegra241_cmdqv_sync_vcmdq (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t *ptr; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONS_INDX ); +if (!ptr) { +return ; + } + *ptr = cmdqv->vcmdq_cons_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_PROD_INDX ); + *ptr = cmdqv->vcmdq_prod_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONFIG ); + *ptr = cmdqv->vcmdq_config[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_GERRORN ); + *ptr = cmdqv->vcmdq_gerrorn[index]; +} +/* * Allocate a host HW VCMDQ from the current cached BASE / size for @index. * @@ -94,6 +127,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, hw_queue->viommu = viommu; cmdqv->vcmdq[index] = hw_queue; +/* Pre-alloc cached writes survive the cache-to-hardware transition. */ +tegra241_cmdqv_sync_vcmdq (cmdqv, index); +return true; }@@ -108,11 +144,14 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv, }/* -* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. +* Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. Reads from the +* mmap'd VINTF page0 when the VCMDQ is allocated (HW is the source of +* truth post-alloc); otherwise reads from the register cache. */ static bool tegra241_vcmdq_enabled (Tegra241CMDQV *cmdqv,int index) { -uint32_t status = cmdqv->vcmdq_status[index]; +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_STATUS ); +uint32_t status = ptr ? *ptr : cmdqv->vcmdq_status[index];return status &R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; }@@ -122,12 +161,21 @@ static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)* * The caller normalizes the MMIO offset such that @offset0 always refers * to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly +* from the VINTF page0 backing. Otherwise, fall back to the cache. */ static uint64_t tegra241_cmdqv_read_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0);uint64_t val =0 ; +if (ptr) { + val = *ptr; + goto out; + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : val = cmdqv->vcmdq_cons_indx[index];@@ -152,6 +200,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_read_vcmdq_page0 (index, offset0, val);return val; }@@ -225,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,* * Page 0 registers are all 32-bit; this helper is only called for 4-byte * writes. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly +* to the VINTF page0 backing. Otherwise, update the cache. */ static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index,uint32_t value) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0); + +if (ptr) { +switch (offset0) { +case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : cmdqv->vcmdq_cons_indx[index] = value;
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 326
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -253,6 +320,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_write_vcmdq_page0 (index, offset0, value); }
Thanks
Eric
RE: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Hi Eric,
Quoted 34 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backingExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 9 lines · Shameer Kolothum
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:
- Direct VCMDQ Page0 aperture (offset
0x10000)
- VINTF Page0 (offset0x30000)Quoted 6 lines
These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.
VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (
IOMMU_HW_QUEUE_ALLOC)
Quoted 7 lines
has
Quoted 3 lines
been allocated for the VCMDQ:
Pre-alloc:
vintf_ptris NULL. Both apertures use QEMU's registerdo you mean
vintf_page0instead ofvintf_ptr.
vintf_page0is set on mmap, right? and not onhwqueue_alloc
Yeah. I actually meant the ptr returned by tegra241_cmdqv_vintf_ptr()
Quoted 4 lines
Besides pre/post-alloc sounds a bit confusing to me since it can mean
queue_allocor mmap.
If I understand correctlyvintf_page0can be set whilehwqueue_allochas not been done yet and conversely, hence the double check below:
The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.
Anyway, I will make it clear.
Quoted 25 lines
+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
Quoted 8 lines
cache. Hardware is not yet engaged.
Post-alloc:
vintf_ptris valid. Both QEMU trapped apertures access
registers directly via the mmap'dvintf_page0pointer, bypassing the cache. Hardware is the single source of truth.The pre-to-post-alloc transition is triggered by the IOMMUFD hardware
so here you mean alloc is hw queue alloc
Quoted 2 lines
queue allocation. The
tegra241_cmdqv_sync_vcmdq()copies any pre-alloc cached writes (CONS_INDX,PROD_INDX, CONFIG, GERRORN) into themmap'd
Quoted 8 lines
page so the guest's view survives the transition.
CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".
Quoted 8 lines
I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32
Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?
Ok. Let me try to explain my understanding.
This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-Nvidia/
Consider the example:
QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:
write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX
If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):
" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any Virtual CMDQ, the access is dropped with no Fault/Interrupt."
So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.
I think this contradicts spec p.176 where it says:
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."
So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.
@Nicolin: did I understand your earlier reasoning correctly here?
Thanks,
Shameer
Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 25 lines · Shameer Kolothum Thodi
QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDXIf we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):
" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any Virtual CMDQ, the access is dropped with no Fault/Interrupt."
So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.
I think this contradicts spec p.176 where it says:
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."
Quoted 4 lines
So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.
@Nicolin: did I understand your earlier reasoning correctly here?
Yes.
Nicolin
Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
In reply to Shameer Kolothum ThodiHi Shameer,
Quoted 100 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 34 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backingExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 9 lines · Shameer Kolothum
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:
- Direct VCMDQ Page0 aperture (offset
0x10000)
- VINTF Page0 (offset0x30000)Quoted 6 lines
These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.
VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (
IOMMU_HW_QUEUE_ALLOC)Quoted 7 lines
has
Quoted 3 lines
been allocated for the VCMDQ:
Pre-alloc:
vintf_ptris NULL. Both apertures use QEMU's registerdo you mean
vintf_page0instead ofvintf_ptr.
vintf_page0is set on mmap, right? and not onhwqueue_allocYeah. I actually meant the ptr returned by
tegra241_cmdqv_vintf_ptr()Quoted 4 lines
Besides pre/post-alloc sounds a bit confusing to me since it can mean
queue_allocor mmap.
If I understand correctlyvintf_page0can be set whilehwqueue_allochas not been done yet and conversely, hence the double check below:The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.
Anyway, I will make it clear.
Quoted 23 lines
+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
Quoted 8 lines
cache. Hardware is not yet engaged.
Post-alloc:
vintf_ptris valid. Both QEMU trapped apertures access
registers directly via the mmap'dvintf_page0pointer, bypassing the cache. Hardware is the single source of truth.The pre-to-post-alloc transition is triggered by the IOMMUFD hardware
so here you mean alloc is hw queue alloc
Quoted 2 lines
queue allocation. The
tegra241_cmdqv_sync_vcmdq()copies any pre-alloc cached writes (CONS_INDX,PROD_INDX, CONFIG, GERRORN) into themmap'd
Quoted 8 lines
page so the guest's view survives the transition.
CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".
Quoted 8 lines
I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32
Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?
Ok. Let me try to explain my understanding.
This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-Nvidia/Consider the example:
QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDX
Quoted 7 lines
If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):
" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any Virtual CMDQ, the access is dropped with no Fault/Interrupt."
OK
Quoted 6 lines
So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.
I think this contradicts spec p.176 where it says:
Quoted 7 lines
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."
So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.
Hum. So it is said if the cmdqv is not mapped on the VI you cannot send commands to it through the direct aperture. But it does not say what does happen if you do so. Why wouldn't the VI behavior be wrong, ie. "
the access is dropped with no Fault/Interrupt."
Eric
Quoted 5 lines
@Nicolin: did I understand your earlier reasoning correctly here?
Thanks,
Shameer
RE: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 144 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:20
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backingExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 13 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'dVINTF
Quoted 5 lines
Quoted 5 lines
page0 as VCMDQ backing
External email: Use caution opening links or attachments
Quoted 8 lines
Quoted 8 lines
Hi Shameer,
Quoted 5 lines · Shameer Kolothum
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.There are two QEMU trapped MMIO apertures for VCMDQ Page0
registers:
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
- Direct VCMDQ Page0 aperture (offset
0x10000)
- VINTF Page0 (offset0x30000)These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.
VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (
IOMMU_HW_QUEUE_ALLOC)Quoted 7 lines
has
Quoted 3 lines
been allocated for the VCMDQ:
Pre-alloc:
vintf_ptris NULL. Both apertures use QEMU's registerdo you mean
vintf_page0instead ofvintf_ptr.
vintf_page0is set on mmap, right? and not onhwqueue_allocYeah. I actually meant the ptr returned by
tegra241_cmdqv_vintf_ptr()Quoted 22 lines
Quoted 4 lines
Besides pre/post-alloc sounds a bit confusing to me since it can mean
queue_allocor mmap.
If I understand correctlyvintf_page0can be set whilehwqueue_allochas not been done yet and conversely, hence the double check below:The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.
Anyway, I will make it clear.
Quoted 13 lines
+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
Quoted 8 lines
cache. Hardware is not yet engaged.
Post-alloc:
vintf_ptris valid. Both QEMU trapped apertures access
registers directly via the mmap'dvintf_page0pointer, bypassing the cache. Hardware is the single source of truth.The pre-to-post-alloc transition is triggered by the IOMMUFD hardware
so here you mean alloc is hw queue alloc
Quoted 1 line
queue allocation. The
tegra241_cmdqv_sync_vcmdq()copies any pre-alloc
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
cached writes (
CONS_INDX,PROD_INDX, CONFIG, GERRORN) into themmap'd
Quoted 8 lines
page so the guest's view survives the transition.
CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".
I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32
Quoted 8 lines
Quoted 3 lines
Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?
Ok. Let me try to explain my understanding.
This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-Nvidia/
Quoted 10 lines
Consider the example:
QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDXQuoted 6 lines
If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):
" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any
Virtual
Quoted 1 line
CMDQ, the access is dropped with no Fault/Interrupt."
OK
Quoted 11 lines
So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.
I think this contradicts spec p.176 where it says:
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."
Quoted 7 lines
Quoted 2 lines
So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.
Hum. So it is said if the cmdqv is not mapped on the VI you cannot send commands to it through the direct aperture. But it does not say what does happen if you do so. Why wouldn't the VI behavior be wrong, ie. "
the access is dropped with no Fault/Interrupt."
"While the software can program the Virtual CMDQ(s) directly using the
direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."
Assuming your question is regarding the above quote:
The quote is about the direct aperture and says the software "can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture", and only the "send commands to the SMMU" step is gated by VINTF allocation. If direct-aperture writes were also dropped for unallocated VCMDQs, the "can program directly" wording wouldn't be meaningful.
"Can program" behaviour:
write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0x10
"Drop" behaviour:
write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0 or undefined
Hope this answers your question. let me know if not.
Thanks,
Shameer
Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 166 lines · Shameer Kolothum Thodi
Quoted 140 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:20
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backingExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 13 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'dVINTF
Quoted 5 lines
Quoted 5 lines
page0 as VCMDQ backing
External email: Use caution opening links or attachments
Quoted 8 lines
Quoted 8 lines
Hi Shameer,
Quoted 5 lines · Shameer Kolothum
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.There are two QEMU trapped MMIO apertures for VCMDQ Page0
registers:
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
- Direct VCMDQ Page0 aperture (offset
0x10000)
- VINTF Page0 (offset0x30000)These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.
VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (
IOMMU_HW_QUEUE_ALLOC)Quoted 7 lines
has
Quoted 3 lines
been allocated for the VCMDQ:
Pre-alloc:
vintf_ptris NULL. Both apertures use QEMU's registerdo you mean
vintf_page0instead ofvintf_ptr.
vintf_page0is set on mmap, right? and not onhwqueue_allocYeah. I actually meant the ptr returned by
tegra241_cmdqv_vintf_ptr()Quoted 22 lines
Quoted 4 lines
Besides pre/post-alloc sounds a bit confusing to me since it can mean
queue_allocor mmap.
If I understand correctlyvintf_page0can be set whilehwqueue_allochas not been done yet and conversely, hence the double check below:The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.
Anyway, I will make it clear.
Quoted 13 lines
+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
Quoted 8 lines
cache. Hardware is not yet engaged.
Post-alloc:
vintf_ptris valid. Both QEMU trapped apertures access
registers directly via the mmap'dvintf_page0pointer, bypassing the cache. Hardware is the single source of truth.The pre-to-post-alloc transition is triggered by the IOMMUFD hardware
so here you mean alloc is hw queue alloc
Quoted 1 line
queue allocation. The
tegra241_cmdqv_sync_vcmdq()copies any pre-alloc
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
cached writes (
CONS_INDX,PROD_INDX, CONFIG, GERRORN) into themmap'd
Quoted 8 lines
page so the guest's view survives the transition.
CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".
I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32
Quoted 8 lines
Quoted 3 lines
Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?
Ok. Let me try to explain my understanding.
This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-Nvidia/
Quoted 9 lines
Consider the example:
QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDXQuoted 6 lines
If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):
" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any
Virtual
Quoted 1 line
CMDQ, the access is dropped with no Fault/Interrupt."
OK
Quoted 10 lines
So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.
I think this contradicts spec p.176 where it says:
"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."
Quoted 7 lines
Quoted 2 lines
So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.
Hum. So it is said if the cmdqv is not mapped on the VI you cannot send commands to it through the direct aperture. But it does not say what does happen if you do so. Why wouldn't the VI behavior be wrong, ie. "
the access is dropped with no Fault/Interrupt."
"While the software can program the Virtual CMDQ(s) directly using the
direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."Assuming your question is regarding the above quote:
The quote is about the direct aperture and says the software "can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture", and only the "send commands to the SMMU" step is gated by VINTF allocation. If direct-aperture writes were also dropped for unallocated VCMDQs, the "can program directly" wording wouldn't be meaningful.
"Can program" behaviour:
write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDX// read returns0x10
Quoted 6 lines
"Drop" behaviour:
write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDX// read returns 0 or undefinedHope this answers your question. let me know if not.
On my end, I read that the software can use either the direct view or the VINTF view as long as the cmdqv is mapped. And if the vcmdq is not mapped we are not really told what happens.
But anyway this use case of using the direct view looks pretty marginal, doesn't it. So feel free to interpret it as you agree with Nicolin and HW experts.
Thanks
Eric
Quoted 8 lines
Thanks,
Shameer
RE: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 44 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 14:53
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backingExternal email: Use caution opening links or attachments
Quoted 25 lines · Shameer Kolothum Thodi
Quoted 1 line
the access is dropped with no Fault/Interrupt."
"While the software can program the Virtual CMDQ(s) directly using the
direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."Assuming your question is regarding the above quote:
The quote is about the direct aperture and says the software "can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture", and only the "send commands to the SMMU" step is gated by VINTF allocation. If direct-aperture writes were also dropped for unallocated VCMDQs, the "can program directly" wording wouldn't be meaningful.
"Can program" behaviour:
write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDX// read returns0x10"Drop" behaviour:
write
0x10->A_VCMDQ1_PROD_INDX
readA_VCMDQ1_PROD_INDX// read returns 0 or undefined
Quoted 7 lines
Quoted 1 line
Hope this answers your question. let me know if not.
On my end, I read that the software can use either the direct view or the VINTF view as long as the cmdqv is mapped. And if the vcmdq is not mapped we are not really told what happens.
But anyway this use case of using the direct view looks pretty marginal, doesn't it.
Yes, the direct aperture read/write isn't an expected common use case.
So feel free to interpret it as you agree with Nicolin and
Quoted 1 line
HW experts.
Thanks. I will keep it as is and have another look at the "overview" section to improve the description of this.
Thanks,
Shameer
PATCH v5 21/32 memory: Allow RAM device regions to skip IOMMU mapping
In reply to Shameer KolothumSome RAM device regions created with memory_region_init_ram_device_ptr() are not intended to be P2P DMA targets.
The VFIO listener currently treats all RAM device regions as DMA capable and attempts to map them into the IOMMU. For regions without dma-buf backing this fails and prints warnings such as:
IOMMU_IOAS_MAP failed: Bad address, PCI BAR?
Introduce a MemoryRegion flag (ram_device_skip_iommu_map) to mark RAM device regions that should not be IOMMU mapped, paired with memory_region_skip_iommu_map() / memory_region_set_skip_iommu_map() accessors. When the flag is set, the VFIO listener skips DMA mapping for that region.
--- include/system/memory.h | 21 +++++++++++++++++++++ hw/vfio/listener.c | 4 ++++ system/memory.c | 10 ++++++++++ 3 files changed, 35 insertions(+)
include/system/memory.h +21 -0
@@ -864,6 +864,8 @@ struct MemoryRegion {/* For devices designed to perform re-entrant IO into their own IO MRs */ bool disable_reentrancy_guard; +/* RAM device region that does not require IOMMU mapping for P2P */ +bool ram_device_skip_iommu_map; };struct IOMMUMemoryRegion {
@@ -1743,6 +1745,25 @@ static inline bool memory_region_is_romd(const MemoryRegion *mr)*/ bool memory_region_is_protected (const MemoryRegion *mr); +/** +* memory_region_skip_iommu_map: check whether a memory region is excluded +* from IOMMU mapping +* +* Returns %true if @mr is a RAM device region marked to skip IOMMU mapping. +* +* @mr: the memory region being queried +*/ +bool memory_region_skip_iommu_map (const MemoryRegion *mr); + +/** +* memory_region_set_skip_iommu_map: mark a RAM device region to skip IOMMU +* mapping +* +* @mr: the memory region being modified +* @skip: %true to skip IOMMU mapping, %false to allow it +*/ +void memory_region_set_skip_iommu_map (MemoryRegion *mr,bool skip); +/** * memory_region_has_guest_memfd: check whether a memory region has guest_memfd * associated
hw/vfio/listener.c +4 -0
@@ -610,6 +610,10 @@ void vfio_container_region_add(VFIOContainer *bcontainer, } } +if (memory_region_skip_iommu_map (section->mr)) { +return ; + } + ret =vfio_container_dma_map (bcontainer, iova,int128_get64 (llsize), vaddr, section->readonly, section->mr);if (ret) {
system/memory.c +10 -0
@@ -1814,6 +1814,16 @@ bool memory_region_is_protected(const MemoryRegion *mr)return mr->ram && (mr->ram_block->flags &RAM_PROTECTED ); } +bool memory_region_skip_iommu_map (const MemoryRegion *mr) +{ +return memory_region_is_ram_device (mr) && mr->ram_device_skip_iommu_map; +} + +void memory_region_set_skip_iommu_map (MemoryRegion *mr,bool skip) +{ + mr->ram_device_skip_iommu_map = skip; +} +bool memory_region_has_guest_memfd (const MemoryRegion *mr) {return mr->ram_block && mr->ram_block->guest_memfd >=0 ;
Signature · 1 line
2.43.0
Re: [PATCH v5 21/32] memory: Allow RAM device regions to skip IOMMU mapping
Hi Shameer,
Quoted 66 lines · Shameer Kolothum
Some RAM device regions created with
memory_region_init_ram_device_ptr()are not intended to be P2P DMA targets.The VFIO listener currently treats all RAM device regions as DMA capable and attempts to map them into the IOMMU. For regions without dma-buf backing this fails and prints warnings such as:
IOMMU_IOAS_MAPfailed: Bad address, PCI BAR?Introduce a MemoryRegion flag (
ram_device_skip_iommu_map) to mark RAM device regions that should not be IOMMU mapped, paired withmemory_region_skip_iommu_map()/memory_region_set_skip_iommu_map()accessors. When the flag is set, the VFIO listener skips DMA mapping for that region.Signed-off-by Shameer Kolothum <[email protected]>--- include/system/memory.h | 21 +++++++++++++++++++++ hw/vfio/listener.c | 4 ++++ system/memory.c | 10 ++++++++++ 3 files changed, 35 insertions(+)include/system/memory.h +21
-0@@ -864,6 +864,8 @@ struct MemoryRegion {/* For devices designed to perform re-entrant IO into their own IO MRs */ bool disable_reentrancy_guard; +/* RAM device region that does not require IOMMU mapping for P2P */ +bool ram_device_skip_iommu_map; };struct IOMMUMemoryRegion {@@ -1743,6 +1745,25 @@ static inline bool memory_region_is_romd(const MemoryRegion *mr)*/ bool memory_region_is_protected (const MemoryRegion *mr); +/** +* memory_region_skip_iommu_map: check whether a memory region is excluded +* from IOMMU mapping +* +* Returns %true if @mr is a RAM device region marked to skip IOMMU mapping. +* +* @mr: the memory region being queried +*/ +bool memory_region_skip_iommu_map (const MemoryRegion *mr); + +/** +* memory_region_set_skip_iommu_map: mark a RAM device region to skip IOMMU +* mapping +* +* @mr: the memory region being modified +* @skip: %true to skip IOMMU mapping, %false to allow it +*/ +void memory_region_set_skip_iommu_map (MemoryRegion *mr,bool skip); +/** * memory_region_has_guest_memfd: check whether a memory region has guest_memfd * associated hw/vfio/listener.c +0
-0
hw/vfio/listener.c void vfio_container_region_add(VFIOContainer *bcontainer, · line 613
Quoted 4 lines
hw/vfio/listener.c +1
-0@@ -610,6 +610,10 @@ void vfio_container_region_add(VFIOContainer *bcontainer, } } +if (memory_region_skip_iommu_map (section->mr)) {
Maybe add a trace-point just as abovetrace_vfio_listener_region_add_no_dma_map()
Otherwise looks good to me.
Thanks
Eric
Quoted 27 lines
hw/vfio/listener.c +3
-0+return ; + } + ret =vfio_container_dma_map (bcontainer, iova,int128_get64 (llsize), vaddr, section->readonly, section->mr);if (ret) {system/memory.c +10
-0@@ -1814,6 +1814,16 @@ bool memory_region_is_protected(const MemoryRegion *mr)return mr->ram && (mr->ram_block->flags &RAM_PROTECTED ); } +bool memory_region_skip_iommu_map (const MemoryRegion *mr) +{ +return memory_region_is_ram_device (mr) && mr->ram_device_skip_iommu_map; +} + +void memory_region_set_skip_iommu_map (MemoryRegion *mr,bool skip) +{ + mr->ram_device_skip_iommu_map = skip; +} +bool memory_region_has_guest_memfd (const MemoryRegion *mr) {return mr->ram_block && mr->ram_block->guest_memfd >=0 ;
PATCH v5 22/32 hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Once a VCMDQ is allocated, map the mmap'd vintf_page0 region directly into the guest-visible MMIO space at offset 0x30000 as a RAM-backed MemoryRegion. This eliminates QEMU trapping for hot-path CONS/PROD index updates.
After this patch, the two VCMDQ apertures use different access paths:
the direct aperture (0x10000) remains QEMU-trapped and writes via vintf_ptr, while the VI aperture (0x30000) is a direct guest RAM mapping. Both paths write to the same underlying vintf_page0 memory, so no synchronisation between the apertures is needed.
The mapping is installed lazily on first successful VCMDQ hardware queue allocation and removed when CMDQV or VINTF is disabled.
--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
hw/arm/tegra241-cmdqv.h +1 -0
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {IOMMUFDVeventq *veventq;IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0; +MemoryRegion *mr_vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;
hw/arm/tegra241-cmdqv.c +39 -0
@@ -18,6 +18,40 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_guest_unmap_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +if (!cmdqv->mr_vintf_page0) { +return ; + } + +memory_region_del_subregion (&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0); +object_unparent (OBJECT (cmdqv->mr_vintf_page0)); +g_free (cmdqv->mr_vintf_page0); + cmdqv->mr_vintf_page0 =NULL ; +} + +static void tegra241_cmdqv_guest_map_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +char *name; + +if (cmdqv->mr_vintf_page0) { +return ; + } + + name =g_strdup_printf ("%s vintf-page0" , +memory_region_name (&cmdqv->mmio_cmdqv)); + cmdqv->mr_vintf_page0 =g_malloc0 (sizeof (*cmdqv->mr_vintf_page0)); +memory_region_init_ram_device_ptr (cmdqv->mr_vintf_page0, +memory_region_owner (&cmdqv->mmio_cmdqv), + name,VINTF_PAGE_SIZE , + cmdqv->vintf_page0); +memory_region_set_skip_iommu_map (cmdqv->mr_vintf_page0, true); +memory_region_add_subregion_overlap (&cmdqv->mmio_cmdqv, +CMDQV_VINTF_PAGE0_BASE , + cmdqv->mr_vintf_page0,1 ); +g_free (name); +} +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) {IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
@@ -130,6 +164,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,/* Pre-alloc cached writes survive the cache-to-hardware transition. */ tegra241_cmdqv_sync_vcmdq (cmdqv, index); +/* Map the mmap'd VINTF page0 into guest MMIO space. */ +tegra241_cmdqv_guest_map_vintf_page0 (cmdqv); +return true; }
@@ -449,6 +486,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,tegra241_cmdqv_setup_all_vcmdq (cmdqv, errp); } }else { +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv);tegra241_cmdqv_free_all_vcmdq (cmdqv);tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ;
@@ -566,6 +604,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,*/ tegra241_cmdqv_setup_all_vcmdq (cmdqv, &local_err); }else { +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv);tegra241_cmdqv_free_all_vcmdq (cmdqv); cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK ; }
Signature · 1 line
2.43.0
Re: [PATCH v5 22/32] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space
Quoted 7 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Once a VCMDQ is allocated, map the mmap'd
vintf_page0region directly into the guest-visible MMIO space at offset0x30000as a RAM-backed MemoryRegion. This eliminates QEMU trapping for hot-path CONS/PROD index updates.
Quoted 5 lines
After this patch, the two VCMDQ apertures use different access paths:
the direct aperture (0x10000) remains QEMU-trapped and writes viavintf_ptr, while the VI aperture (0x30000) is a direct guest RAM mapping. Both paths write to the same underlyingvintf_page0memory, so no synchronisation between the apertures is needed.
Please further explain the rationale of having 2 separate paths
Eric
Quoted 80 lines
The mapping is installed lazily on first successful VCMDQ hardware queue allocation and removed when CMDQV or VINTF is disabled.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)hw/arm/tegra241-cmdqv.h +1
-0@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {IOMMUFDVeventq *veventq;IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0; +MemoryRegion *mr_vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c +37
-0@@ -18,6 +18,40 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_guest_unmap_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +if (!cmdqv->mr_vintf_page0) { +return ; + } + +memory_region_del_subregion (&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0); +object_unparent (OBJECT (cmdqv->mr_vintf_page0)); +g_free (cmdqv->mr_vintf_page0); + cmdqv->mr_vintf_page0 =NULL ; +} + +static void tegra241_cmdqv_guest_map_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +char *name; + +if (cmdqv->mr_vintf_page0) { +return ; + } + + name =g_strdup_printf ("%s vintf-page0" , +memory_region_name (&cmdqv->mmio_cmdqv)); + cmdqv->mr_vintf_page0 =g_malloc0 (sizeof (*cmdqv->mr_vintf_page0)); +memory_region_init_ram_device_ptr (cmdqv->mr_vintf_page0, +memory_region_owner (&cmdqv->mmio_cmdqv), + name,VINTF_PAGE_SIZE , + cmdqv->vintf_page0); +memory_region_set_skip_iommu_map (cmdqv->mr_vintf_page0, true); +memory_region_add_subregion_overlap (&cmdqv->mmio_cmdqv, +CMDQV_VINTF_PAGE0_BASE , + cmdqv->mr_vintf_page0,1 ); +g_free (name); +} +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) {IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;@@ -130,6 +164,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,/* Pre-alloc cached writes survive the cache-to-hardware transition. */ tegra241_cmdqv_sync_vcmdq (cmdqv, index); +/* Map the mmap'd VINTF page0 into guest MMIO space. */ +tegra241_cmdqv_guest_map_vintf_page0 (cmdqv); +return true; }
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 486
Quoted 1 line
hw/arm/tegra241-cmdqv.c +0
-0@@ -449,6 +486,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,tegra241_cmdqv_setup_all_vcmdq (cmdqv, errp);
I would rather put the tegra241_cmdqv_guest_map_vintf_page0(cmdqv) here to have symetric map/unmap
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 605
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +1
-0} }else { +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv);tegra241_cmdqv_free_all_vcmdq (cmdqv);tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ;@@ -566,6 +604,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,*/ tegra241_cmdqv_setup_all_vcmdq (cmdqv, &local_err);
same
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 610
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1
-0}else { +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv);tegra241_cmdqv_free_all_vcmdq (cmdqv); cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK ; }
Thanks
Eric
PATCH v5 23/32 hw/arm/smmuv3-accel: Introduce common helper for veventq read
In reply to Shameer KolothumMove the vEVENTQ read and validation logic into a common helper smmuv3_accel_event_read_validate(). The helper performs the read(), checks for overflow and short reads, validates the sequence number, and updates the sequence state.
This helper can be reused for Tegra241 CMDQV vEVENTQ support in a subsequent patch.
Error handling is slightly adjusted: instead of reporting errors directly in the read handler, the helper now returns errors via Error **. Sequence gaps are reported as warnings.
--- hw/arm/smmuv3-accel.h | 2 ++ hw/arm/smmuv3-accel-stubs.c | 7 ++++ hw/arm/smmuv3-accel.c | 67 ++++++++++++++++++++++--------------- 3 files changed, 49 insertions(+), 27 deletions(-)
hw/arm/smmuv3-accel.h +2 -0
@@ -88,6 +88,8 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *s, void *cmd, SMMUDevice *sdev, Error **errp);void smmuv3_accel_idr_override (SMMUv3State *s);bool smmuv3_accel_alloc_veventq (SMMUv3State *s,Error **errp); +bool smmuv3_accel_event_read_validate (IOMMUFDVeventq *veventq,uint32_t type, +void *buf,size_t size,Error **errp);void smmuv3_accel_reset (SMMUv3State *s);#endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +7 -0
@@ -47,6 +47,13 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)return true; } +bool smmuv3_accel_event_read_validate (IOMMUFDVeventq *veventq,uint32_t type, +void *buf,size_t size,Error **errp) +{ +return true; +} + +void smmuv3_accel_reset (SMMUv3State *s) { }
hw/arm/smmuv3-accel.c +40 -27
87 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v5 24/32 hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
In reply to Shameer KolothumInstall an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.
The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().
--- hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 61 insertions(+)
hw/arm/tegra241-cmdqv.c +60 -0
@@ -14,6 +14,7 @@#include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" +#include "hw/core/irq.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" #include "trace.h"
@@ -736,6 +737,48 @@ out:trace_tegra241_cmdqv_write_mmio (offset, value, size); } +static void tegra241_cmdqv_event_read (void *opaque) +{ +Tegra241CMDQV *cmdqv = opaque; +IOMMUFDVeventq *veventq = cmdqv->veventq; +struct { +struct iommufd_vevent_header hdr; +struct iommu_vevent_tegra241_cmdqv vevent; + } buf; +Error *local_err =NULL ; + +if (!smmuv3_accel_event_read_validate (veventq, +IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV , + &buf,sizeof (buf), &local_err)) { +warn_report_err_once (local_err); +return ; + } + +if (buf.vevent.lvcmdq_err_map[0 ] || buf.vevent.lvcmdq_err_map[1 ]) { + cmdqv->vintf_cmdq_err_map[0 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[1 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],32 ,32 ); + cmdqv->vintf_cmdq_err_map[2 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[3 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],32 ,32 ); +/* +* CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct +* registers (different MMIO offsets). With only VINTF0 exposed +* they carry the same data, so mirror. +*/ +for (int i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i]; + } + cmdqv->vi_err_map[0 ] |=0x1 ; +qemu_irq_pulse (cmdqv->irq); +trace_tegra241_cmdqv_err_map ( + cmdqv->vintf_cmdq_err_map[3 ], cmdqv->vintf_cmdq_err_map[2 ], + cmdqv->vintf_cmdq_err_map[1 ], cmdqv->vintf_cmdq_err_map[0 ]); + } +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) {SMMUv3AccelState *accel = s->s_accel;
@@ -747,6 +790,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)return ; }if (veventq) { +qemu_set_fd_handler (veventq->veventq_fd,NULL ,NULL ,NULL );close (veventq->veventq_fd);iommufd_backend_free_id (viommu->iommufd, veventq->veventq_id);g_free (veventq);
@@ -762,6 +806,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;uint32_t viommu_id, veventq_id, veventq_fd;IOMMUFDVeventq *veventq; +int flags; if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid, IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
@@ -780,14 +825,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, goto free_viommu; } + flags =fcntl (veventq_fd,F_GETFL ); +if (flags <0 ) { +error_setg (errp,"Failed to get flags for vEVENTQ fd" ); + goto free_veventq; + } +if (fcntl (veventq_fd,F_SETFL ,O_NONBLOCK | flags) <0 ) { +error_setg (errp,"Failed to set O_NONBLOCK on vEVENTQ fd" ); + goto free_veventq; + } + veventq =g_new (IOMMUFDVeventq,1 ); veventq->veventq_id = veventq_id; veventq->veventq_fd = veventq_fd; cmdqv->veventq = veventq; +/* Set up event handler for veventq fd */ +qemu_set_fd_handler (veventq_fd, tegra241_cmdqv_event_read,NULL , cmdqv); *out_viommu_id = viommu_id;return true; +free_veventq: +close (veventq_fd); +iommufd_backend_free_id (idev->iommufd, veventq_id); free_viommu:iommufd_backend_free_id (idev->iommufd, viommu_id);return false;
hw/arm/trace-events +1 -0
@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
Hi Shameer,
Quoted 55 lines · Shameer Kolothum
Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.
The handler runs in QEMU's main loop, using a non-blocking fd registered via
qemu_set_fd_handler().Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 61 insertions(+)hw/arm/tegra241-cmdqv.c +27
-0@@ -14,6 +14,7 @@#include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" +#include "hw/core/irq.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" #include "trace.h" @@ -736,6 +737,48 @@ out:trace_tegra241_cmdqv_write_mmio (offset, value, size); } +static void tegra241_cmdqv_event_read (void *opaque) +{ +Tegra241CMDQV *cmdqv = opaque; +IOMMUFDVeventq *veventq = cmdqv->veventq; +struct { +struct iommufd_vevent_header hdr; +struct iommu_vevent_tegra241_cmdqv vevent; + } buf; +Error *local_err =NULL ; + +if (!smmuv3_accel_event_read_validate (veventq, +IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV , + &buf,sizeof (buf), &local_err)) { +warn_report_err_once (local_err); +return ; + } + +if (buf.vevent.lvcmdq_err_map[0 ] || buf.vevent.lvcmdq_err_map[1 ]) { + cmdqv->vintf_cmdq_err_map[0 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[1 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],32 ,32 ); + cmdqv->vintf_cmdq_err_map[2 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[3 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],32 ,32 );
hw/arm/tegra241-cmdqv.c out: · line 775
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+/* +* CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct +* registers (different MMIO offsets). With only VINTF0 exposed +* they carry the same data, so mirror. +*/ +for (int i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i]; + } + cmdqv->vi_err_map[0 ] |=0x1 ; +qemu_irq_pulse (cmdqv->irq);
I don't see SMMU_CMDQV_VI_ERR_MAP_0 updated anywhere. If I understand correctly, if you get an error on any VCMDQ you should update it. Also the triggering of the irq should depnd on SMMU_CMDQ_VI_INT_MASK0, no?
Quoted 60 lines
hw/arm/tegra241-cmdqv.c +23
-0+trace_tegra241_cmdqv_err_map ( + cmdqv->vintf_cmdq_err_map[3 ], cmdqv->vintf_cmdq_err_map[2 ], + cmdqv->vintf_cmdq_err_map[1 ], cmdqv->vintf_cmdq_err_map[0 ]); + } +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) {SMMUv3AccelState *accel = s->s_accel;@@ -747,6 +790,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)return ; }if (veventq) { +qemu_set_fd_handler (veventq->veventq_fd,NULL ,NULL ,NULL );close (veventq->veventq_fd);iommufd_backend_free_id (viommu->iommufd, veventq->veventq_id);g_free (veventq);@@ -762,6 +806,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;uint32_t viommu_id, veventq_id, veventq_fd;IOMMUFDVeventq *veventq; +int flags; if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid, IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,@@ -780,14 +825,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, goto free_viommu; } + flags =fcntl (veventq_fd,F_GETFL ); +if (flags <0 ) { +error_setg (errp,"Failed to get flags for vEVENTQ fd" ); + goto free_veventq; + } +if (fcntl (veventq_fd,F_SETFL ,O_NONBLOCK | flags) <0 ) { +error_setg (errp,"Failed to set O_NONBLOCK on vEVENTQ fd" ); + goto free_veventq; + } + veventq =g_new (IOMMUFDVeventq,1 ); veventq->veventq_id = veventq_id; veventq->veventq_fd = veventq_fd; cmdqv->veventq = veventq; +/* Set up event handler for veventq fd */ +qemu_set_fd_handler (veventq_fd, tegra241_cmdqv_event_read,NULL , cmdqv); *out_viommu_id = viommu_id;return true; +free_veventq: +close (veventq_fd); +iommufd_backend_free_id (idev->iommufd, veventq_id); free_viommu:iommufd_backend_free_id (idev->iommufd, viommu_id);return false;hw/arm/trace-events +0
-0
hw/arm/trace-events smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS · line 81
Quoted 7 lines
hw/arm/trace-events +1
-0@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Thanks
Eric
RE: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
Hi Eric,
Quoted 86 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:27
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errorsExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 10 lines · Shameer Kolothum
Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.
The handler runs in QEMU's main loop, using a non-blocking fd registered via
qemu_set_fd_handler().Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 60+++++++++++++++++++++++++++++++++++++++++
Quoted 37 lines
hw/arm/trace-events | 1 + 2 files changed, 61 insertions(+)hw/arm/tegra241-cmdqv.c +19
-0@@ -14,6 +14,7 @@#include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" +#include "hw/core/irq.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" #include "trace.h" @@ -736,6 +737,48 @@ out:trace_tegra241_cmdqv_write_mmio (offset, value, size); } +static void tegra241_cmdqv_event_read (void *opaque) +{ +Tegra241CMDQV *cmdqv = opaque; +IOMMUFDVeventq *veventq = cmdqv->veventq; +struct { +struct iommufd_vevent_header hdr; +struct iommu_vevent_tegra241_cmdqv vevent; + } buf; +Error *local_err =NULL ; + +if (!smmuv3_accel_event_read_validate (veventq, +IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV , + &buf,sizeof (buf), &local_err)) { +warn_report_err_once (local_err); +return ; + } + +if (buf.vevent.lvcmdq_err_map[0 ] || buf.vevent.lvcmdq_err_map[1 ]) {hw/arm/tegra241-cmdqv.c out: · line 767
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ cmdqv->vintf_cmdq_err_map[0 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[1 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],32 ,32 ); + cmdqv->vintf_cmdq_err_map[2 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[3 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],32 ,32 ); +/* +* CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct
hw/arm/tegra241-cmdqv.c out: · line 775
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+* registers (different MMIO offsets). With only VINTF0 exposed +* they carry the same data, so mirror. +*/ +for (int i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i]; + } + cmdqv->vi_err_map[0 ] |=0x1 ; +qemu_irq_pulse (cmdqv->irq);
Quoted 3 lines
I don't see
SMMU_CMDQV_VI_ERR_MAP_0updated anywhere. If I understand
correctly, if you get an error on any VCMDQ you should update it.
This one above does that:
cmdqv->vi_err_map[0] |= 0x1; it sets bit 0, which is the error bit for VINTF0 (the only VINTF QEMU exposes).
I will add a comment to make it clear.
Also
Quoted 1 line
the triggering of the irq should depnd on
SMMU_CMDQ_VI_INT_MASK0, no?
Right. will add:
if (!(cmdqv->vi_int_mask[0] & BIT(0))) {
qemu_irq_pulse(cmdqv->irq);
}Thanks,
Shameer
Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
Quoted 88 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 86 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:27
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errorsExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 10 lines · Shameer Kolothum
Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.
The handler runs in QEMU's main loop, using a non-blocking fd registered via
qemu_set_fd_handler().Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 60+++++++++++++++++++++++++++++++++++++++++
Quoted 37 lines
hw/arm/trace-events | 1 + 2 files changed, 61 insertions(+)hw/arm/tegra241-cmdqv.c +19
-0@@ -14,6 +14,7 @@#include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" +#include "hw/core/irq.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" #include "trace.h" @@ -736,6 +737,48 @@ out:trace_tegra241_cmdqv_write_mmio (offset, value, size); } +static void tegra241_cmdqv_event_read (void *opaque) +{ +Tegra241CMDQV *cmdqv = opaque; +IOMMUFDVeventq *veventq = cmdqv->veventq; +struct { +struct iommufd_vevent_header hdr; +struct iommu_vevent_tegra241_cmdqv vevent; + } buf; +Error *local_err =NULL ; + +if (!smmuv3_accel_event_read_validate (veventq, +IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV , + &buf,sizeof (buf), &local_err)) { +warn_report_err_once (local_err); +return ; + } + +if (buf.vevent.lvcmdq_err_map[0 ] || buf.vevent.lvcmdq_err_map[1 ]) {hw/arm/tegra241-cmdqv.c out: · line 767
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ cmdqv->vintf_cmdq_err_map[0 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[1 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],32 ,32 ); + cmdqv->vintf_cmdq_err_map[2 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[3 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],32 ,32 ); +/* +* CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+* registers (different MMIO offsets). With only VINTF0 exposed +* they carry the same data, so mirror. +*/ +for (int i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i]; + } + cmdqv->vi_err_map[0 ] |=0x1 ; +qemu_irq_pulse (cmdqv->irq);
Quoted 8 lines
Quoted 3 lines
I don't see
SMMU_CMDQV_VI_ERR_MAP_0updated anywhere. If I understand
correctly, if you get an error on any VCMDQ you should update it.This one above does that:
cmdqv->vi_err_map[0] |=0x1; it sets bit 0, which is the error bit for VINTF0 (the only VINTF QEMU exposes).I will add a comment to make it clear.
Ah OK. Let's add a comment then.
Eric
Quoted 12 lines
Also
Quoted 1 line
the triggering of the irq should depnd on
SMMU_CMDQ_VI_INT_MASK0, no?Right. will add:
if (!(cmdqv->vi_int_mask[0] & BIT(0))) { qemu_irq_pulse(cmdqv->irq); }Thanks,
Shameer
PATCH v5 25/32 hw/arm/tegra241-cmdqv: Add reset handler
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.
--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 48 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 3 files changed, 52 insertions(+)
hw/arm/tegra241-cmdqv.h +3 -0
@@ -87,6 +87,9 @@ FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8) FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)FIELD (CONFIG ,CONS_DRAM_EN ,20 ,1 ) +/* CMDQV_EN=1, PER_CMD_OFFSET=16B, CLK_BATCH=256, CMD_BATCH=32. */ +#define V_CONFIG_RESET 0x00020083 + REG32(PARAM, 0x4) FIELD(PARAM, CMDQV_VER, 0, 4)FIELD (PARAM ,CMDQV_NUM_CMDQ_LOG2 ,4 ,4 )
hw/arm/tegra241-cmdqv.c +48 -0
56 lines not on this page · raw message
hw/arm/trace-events +1 -0
@@ -76,6 +76,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" +tegra241_cmdqv_init_regs(uint32_t param) "hw info received. param: 0x%04X" tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
Re: [PATCH v5 25/32] hw/arm/tegra241-cmdqv: Add reset handler
Hi Shameer,
Quoted 93 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.
Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 48 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 3 files changed, 52 insertions(+)hw/arm/tegra241-cmdqv.h +3
-0@@ -87,6 +87,9 @@ FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8) FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)FIELD (CONFIG ,CONS_DRAM_EN ,20 ,1 ) +/* CMDQV_EN=1, PER_CMD_OFFSET=16B, CLK_BATCH=256, CMD_BATCH=32. */ +#define V_CONFIG_RESET 0x00020083 + REG32(PARAM, 0x4) FIELD(PARAM, CMDQV_VER, 0, 4)FIELD (PARAM ,CMDQV_NUM_CMDQ_LOG2 ,4 ,4 )hw/arm/tegra241-cmdqv.c +48
-0@@ -853,8 +853,56 @@ free_viommu:return false; } +static void tegra241_cmdqv_init_regs (SMMUv3State *s,Tegra241CMDQV *cmdqv) +{ +int i; + + cmdqv->config =V_CONFIG_RESET ; + cmdqv->param =FIELD_DP32 (0 ,PARAM ,CMDQV_VER ,CMDQV_VER ); + cmdqv->param =FIELD_DP32 (cmdqv->param,PARAM ,CMDQV_NUM_CMDQ_LOG2 , +CMDQV_NUM_CMDQ_LOG2 ); + cmdqv->param =FIELD_DP32 (cmdqv->param,PARAM ,CMDQV_NUM_SID_PER_VI_LOG2 , +CMDQV_NUM_SID_PER_VI_LOG2 ); +trace_tegra241_cmdqv_init_regs (cmdqv->param); + cmdqv->status =R_STATUS_CMDQV_ENABLED_MASK ; + +for (i =0 ; i <2 ; i++) { + cmdqv->vi_err_map[i] =0 ; + cmdqv->vi_int_mask[i] =0 ; + } +for (i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] =0 ; + cmdqv->vintf_cmdq_err_map[i] =0 ; + } + cmdqv->vintf_config =0 ; + cmdqv->vintf_status =0 ; +for (i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { + cmdqv->cmdq_alloc_map[i] =0 ; + cmdqv->vcmdq_cons_indx[i] =0 ; + cmdqv->vcmdq_prod_indx[i] =0 ; + cmdqv->vcmdq_config[i] =0 ; + cmdqv->vcmdq_status[i] =0 ; + cmdqv->vcmdq_gerror[i] =0 ; + cmdqv->vcmdq_gerrorn[i] =0 ; + cmdqv->vcmdq_base[i] =0 ; + cmdqv->vcmdq_cons_indx_base[i] =0 ; + } +} +static void tegra241_cmdqv_reset (SMMUv3State *s) { +SMMUv3AccelState *accel = s->s_accel; +Tegra241CMDQV *cmdqv = accel->cmdqv; + +if (!cmdqv) { +return ; + } + +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv); +tegra241_cmdqv_munmap_vintf_page0 (cmdqv,NULL ); +tegra241_cmdqv_free_all_vcmdq (cmdqv); + +tegra241_cmdqv_init_regs (s, cmdqv); }static const MemoryRegionOps mmio_cmdqv_ops = {hw/arm/trace-events +0
-0
hw/arm/trace-events smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS · line 79
Quoted 4 lines
hw/arm/trace-events +1
-0@@ -76,6 +76,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" +tegra241_cmdqv_init_regs(uint32_t param) "hw info received. param: 0x%04X"
why 'hw info received'.
Besides
Eric
Quoted 3 lines
hw/arm/trace-events +0
-0tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
PATCH v5 26/32 system/physmem: Add qemu_ram_backend_pagesize_min() helper
In reply to Shameer KolothumSome hardware accelerators DMA directly to the host physical address of a guest-resident buffer (e.g. set up via IOMMUFD). The HPA is contiguous only within a single host backing page; a buffer larger than the smallest backing page in use may span non-contiguous host pages and fault.
Introduce qemu_ram_backend_pagesize_min(), which returns the smallest page size among RAM regions backed by a memory-backend object. Falls back to qemu_real_host_page_size() if no memory-backend RAM blocks are found. Callers can use this to bound the size of buffers that must remain host-physically contiguous.
A subsequent Tegra241 CMDQV support patch will use this to bound the command queue size exposed to the guest.
--- include/system/ramblock.h | 1 + system/physmem.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+)
include/system/ramblock.h +1 -0
@@ -175,6 +175,7 @@ int qemu_ram_get_fd(const RAMBlock *rb);size_t qemu_ram_pagesize (const RAMBlock *block);size_t qemu_ram_pagesize_largest (void ); +size_t qemu_ram_backend_pagesize_min (void );#include "exec/target_page.h" #include "exec/hwaddr.h"
system/physmem.c +29 -0
@@ -1999,6 +1999,35 @@ size_t qemu_ram_pagesize_largest(void)return largest; } +/* +* Returns the smallest page size among RAM regions backed by a +* memory-backend object. Falls back to qemu_real_host_page_size() if no +* memory-backend RAM blocks are found. +*/ +size_t qemu_ram_backend_pagesize_min (void ) +{ +RAMBlock *rb; +size_t pg, min_pg =SIZE_MAX ; + +RAMBLOCK_FOREACH (rb) { +MemoryRegion *mr = rb->mr; + +if (!mr || !memory_region_is_ram (mr)) { +continue ; + } +if (!object_dynamic_cast (mr->owner,TYPE_MEMORY_BACKEND )) { +continue ; + } + + pg =qemu_ram_pagesize (rb); +if (pg && pg < min_pg) { + min_pg = pg; + } + } + +return (min_pg ==SIZE_MAX ) ?qemu_real_host_page_size () : min_pg; +} +static int memory_try_enable_merging (void *addr,size_t len) {if (!machine_mem_merge (current_machine)) {
Signature · 1 line
2.43.0
Re: [PATCH v5 26/32] system/physmem: Add qemu_ram_backend_pagesize_min() helper
Quoted 1 line · Shameer Kolothum
Some hardware accelerators DMA directly to the host physical address
s/DMA/perform DMA accesses
Quoted 7 lines
of a guest-resident buffer (e.g. set up via IOMMUFD). The HPA is contiguous only within a single host backing page; a buffer larger than the smallest backing page in use may span non-contiguous host pages and fault.
Introduce
qemu_ram_backend_pagesize_min(), which returns the smallest page size among RAM regions backed by a memory-backend object. Falls
Don't you have means to be more precise? I mean you know the GPA of the cmdq, can't you make sure it belongs to a specific ramblock and check its min page_size?
By the way there is find_min_backend_pagesize() that exists. Can't you reuse that?
Thanks
Eric
Quoted 67 lines
back to
qemu_real_host_page_size()if no memory-backend RAM blocks are found. Callers can use this to bound the size of buffers that must remain host-physically contiguous.A subsequent Tegra241 CMDQV support patch will use this to bound the command queue size exposed to the guest.
Cc Peter Xu <[email protected]>Cc "Philippe Mathieu-Daudé" <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- include/system/ramblock.h | 1 + system/physmem.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+)include/system/ramblock.h +1
-0@@ -175,6 +175,7 @@ int qemu_ram_get_fd(const RAMBlock *rb);size_t qemu_ram_pagesize (const RAMBlock *block);size_t qemu_ram_pagesize_largest (void ); +size_t qemu_ram_backend_pagesize_min (void );#include "exec/target_page.h" #include "exec/hwaddr.h" system/physmem.c +29
-0@@ -1999,6 +1999,35 @@ size_t qemu_ram_pagesize_largest(void)return largest; } +/* +* Returns the smallest page size among RAM regions backed by a +* memory-backend object. Falls back to qemu_real_host_page_size() if no +* memory-backend RAM blocks are found. +*/ +size_t qemu_ram_backend_pagesize_min (void ) +{ +RAMBlock *rb; +size_t pg, min_pg =SIZE_MAX ; + +RAMBLOCK_FOREACH (rb) { +MemoryRegion *mr = rb->mr; + +if (!mr || !memory_region_is_ram (mr)) { +continue ; + } +if (!object_dynamic_cast (mr->owner,TYPE_MEMORY_BACKEND )) { +continue ; + } + + pg =qemu_ram_pagesize (rb); +if (pg && pg < min_pg) { + min_pg = pg; + } + } + +return (min_pg ==SIZE_MAX ) ?qemu_real_host_page_size () : min_pg; +} +static int memory_try_enable_merging (void *addr,size_t len) {if (!machine_mem_merge (current_machine)) {
RE: [PATCH v5 26/32] system/physmem: Add qemu_ram_backend_pagesize_min() helper
Quoted 27 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 10:31
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]; Peter Xu <[email protected]>; Philippe Mathieu-Daudé <[email protected]>
Subject: Re: [PATCH v5 26/32] system/physmem: Addqemu_ram_backend_pagesize_min()helperExternal email: Use caution opening links or attachments
Quoted 1 line · Shameer Kolothum
Some hardware accelerators DMA directly to the host physical address
s/DMA/perform DMA accesses
Quoted 5 lines
of a guest-resident buffer (e.g. set up via IOMMUFD). The HPA is contiguous only within a single host backing page; a buffer larger than the smallest backing page in use may span non-contiguous host pages and fault.
Quoted 5 lines
Quoted 2 lines
Introduce
qemu_ram_backend_pagesize_min(), which returns the smallest page size among RAM regions backed by a memory-backend object. FallsDon't you have means to be more precise? I mean you know the GPA of the cmdq, can't you make sure it belongs to a specific ramblock and check its min
page_size?
As you noted in the next patch we don’t have GPA at the time of init.
Quoted 3 lines
By the way there is
find_min_backend_pagesize()that exists. Can't you reuse that?
Ah.. Thanks!. That looks like is doing the same thing. And we also have warpper that uses it, qemu_minrampagesize().
I will get rid of this patch and use that instead.
Thanks,
Shameer
PATCH v5 27/32 hw/arm/tegra241-cmdqv: Limit queue size based on backend page size
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
CMDQV HW reads guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.
Use qemu_ram_backend_pagesize_min() to find the smallest memory- backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.
--- hw/arm/tegra241-cmdqv.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
hw/arm/tegra241-cmdqv.c +17 -0
@@ -16,6 +16,8 @@#include "hw/arm/smmuv3-common.h" #include "hw/core/irq.h" #include "smmuv3-accel.h" +#include "smmuv3-internal.h" +#include "system/ramblock.h" #include "tegra241-cmdqv.h" #include "trace.h"
@@ -856,6 +858,8 @@ free_viommu:static void tegra241_cmdqv_init_regs (SMMUv3State *s,Tegra241CMDQV *cmdqv) {int i; +size_t pgsize; +uint32_t val; cmdqv->config =V_CONFIG_RESET ; cmdqv->param =FIELD_DP32 (0 ,PARAM ,CMDQV_VER ,CMDQV_VER );
@@ -887,6 +891,19 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) cmdqv->vcmdq_base[i] =0 ; cmdqv->vcmdq_cons_indx_base[i] =0 ; } + +/* +* CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the +* queue fits entirely within the smallest backend page size, ensuring +* the command queue is physically contiguous in host memory. +* +* IDR1.CMDQS = log2(max_qsz) - entry_shift +* +* where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4). +*/ + pgsize =qemu_ram_backend_pagesize_min (); + val =FIELD_EX32 (s->idr[1 ],IDR1 ,CMDQS ); + s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,CMDQS ,MIN (ctz64 (pgsize) -4 , val)); }static void tegra241_cmdqv_reset (SMMUv3State *s)
Signature · 1 line
2.43.0
Re: [PATCH v5 27/32] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size
Hi Shameer,
Quoted 47 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
CMDQV HW reads guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.
Use
qemu_ram_backend_pagesize_min()to find the smallest memory- backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)hw/arm/tegra241-cmdqv.c +5
-0@@ -16,6 +16,8 @@#include "hw/arm/smmuv3-common.h" #include "hw/core/irq.h" #include "smmuv3-accel.h" +#include "smmuv3-internal.h" +#include "system/ramblock.h" #include "tegra241-cmdqv.h" #include "trace.h" @@ -856,6 +858,8 @@ free_viommu:static void tegra241_cmdqv_init_regs (SMMUv3State *s,Tegra241CMDQV *cmdqv) {int i; +size_t pgsize; +uint32_t val; cmdqv->config =V_CONFIG_RESET ; cmdqv->param =FIELD_DP32 (0 ,PARAM ,CMDQV_VER ,CMDQV_VER );@@ -887,6 +891,19 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) cmdqv->vcmdq_base[i] =0 ; cmdqv->vcmdq_cons_indx_base[i] =0 ; } +
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) · line 904
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+/* +* CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the +* queue fits entirely within the smallest backend page size, ensuring +* the command queue is physically contiguous in host memory. +* +* IDR1.CMDQS = log2(max_qsz) - entry_shift +* +* where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4). +*/ + pgsize =qemu_ram_backend_pagesize_min ();
OK I see this is done at init size before we know the GPA. But I meant you may end up with a page size that is suboptimal, no?
Could you elaborate on the actual requirements with regards to huge page usage in this patch and in the cover letter?
Thanks
Eric
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +2
-0+ val =FIELD_EX32 (s->idr[1 ],IDR1 ,CMDQS ); + s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,CMDQS ,MIN (ctz64 (pgsize) -4 , val)); }static void tegra241_cmdqv_reset (SMMUv3State *s)
RE: [PATCH v5 27/32] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size
Quoted 81 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 10:35
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 27/32] hw/arm/tegra241-cmdqv: Limit queue size based on backend page sizeExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 34 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
CMDQV HW reads guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.
Use
qemu_ram_backend_pagesize_min()to find the smallest memory- backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)hw/arm/tegra241-cmdqv.c +2
-0@@ -16,6 +16,8 @@#include "hw/arm/smmuv3-common.h" #include "hw/core/irq.h" #include "smmuv3-accel.h" +#include "smmuv3-internal.h" +#include "system/ramblock.h" #include "tegra241-cmdqv.h" #include "trace.h" hw/arm/tegra241-cmdqv.c free_viommu: · line 858
Quoted 1 line
hw/arm/tegra241-cmdqv.c +0
-0@@ -856,6 +858,8 @@ free_viommu: static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV*cmdqv)
hw/arm/tegra241-cmdqv.c static void
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +2
-0{int i; +size_t pgsize; +uint32_t val; cmdqv->config =V_CONFIG_RESET ; cmdqv->param =FIELD_DP32 (0 ,PARAM ,CMDQV_VER ,CMDQV_VER );@@ -887,6 +891,19 @@ static void
tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +3
-0cmdqv->vcmdq_base[i] =0 ; cmdqv->vcmdq_cons_indx_base[i] =0 ; } + +/* +* CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) · line 904
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+* queue fits entirely within the smallest backend page size, ensuring +* the command queue is physically contiguous in host memory. +* +* IDR1.CMDQS = log2(max_qsz) - entry_shift +* +* where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4). +*/ + pgsize =qemu_ram_backend_pagesize_min ();
Quoted 2 lines
OK I see this is done at init size before we know the GPA. But I meant you may end up with a page size that is suboptimal, no?
Yes. The recommended CMDQV queue size is the max 8M and we recommend using hugepages to achieve that.
Quoted 2 lines
Could you elaborate on the actual requirements with regards to huge page usage in this patch and in the cover letter?
Sure. I will add that info.
Thanks,
Shameer
PATCH v5 28/32 hw/arm/smmuv3: Add per-device identifier property
In reply to Shameer KolothumAdd an "identifier" property to the SMMUv3 device and use it when building the ACPI IORT SMMUv3 node Identifier field.
This avoids relying on device enumeration order and provides a stable per-device identifier. A subsequent patch will use the same identifier when generating the DSDT description for Tegra241 CMDQV, ensuring that the IORT and DSDT entries refer to the same SMMUv3 instance.
The identifier is assigned at pre-plug time, accounting for the ITS Group node that build_iort() places before SMMUv3 nodes in the IORT table, so that identifiers are globally unique across all IORT nodes.
No functional change: IORT blob content for bios-tables qtest is identical to before.
--- include/hw/arm/smmuv3.h | 1 + hw/arm/smmuv3.c | 2 ++ hw/arm/virt-acpi-build.c | 5 ++++- hw/arm/virt.c | 12 ++++++++++++ 4 files changed, 19 insertions(+), 1 deletion(-)
include/hw/arm/smmuv3.h +1 -0
@@ -64,6 +64,7 @@ struct SMMUv3State {qemu_irq irq[4 ];QemuMutex mutex;char *stage; +uint8_t identifier;/* SMMU has HW accelerator support for nested S1 + s2 */ bool accel;
hw/arm/smmuv3.c +2 -0
@@ -2126,6 +2126,8 @@ static const Property smmuv3_properties[] = {* Defaults to stage 1 */ DEFINE_PROP_STRING ("stage" , SMMUv3State, stage), +/* Identifier used for ACPI IORT SMMUv3 (and DSDT for CMDQV) generation */ +DEFINE_PROP_UINT8 ("identifier" , SMMUv3State, identifier,0 ),DEFINE_PROP_BOOL ("accel" , SMMUv3State, accel, false),/* GPA of MSI doorbell, for SMMUv3 accel use. */ DEFINE_PROP_UINT64("msi-gpa", SMMUv3State, msi_gpa, 0),
hw/arm/virt-acpi-build.c +4 -1
@@ -349,6 +349,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b) typedef struct AcpiIortSMMUv3Dev { int irq; hwaddr base; + uint8_t id; GArray *rc_smmu_idmaps; /* Offset of the SMMUv3 IORT Node relative to the start of the IORT */ size_t offset;
@@ -411,6 +412,7 @@ static int populate_smmuv3_dev(VirtMachineState *vms, GArray *sdev_blob) &error_abort)); sdev.accel =object_property_get_bool (obj,"accel" , &error_abort); sdev.ats =smmuv3_ats_enabled (ARM_SMMUV3 (obj)); + sdev.id =object_property_get_uint (obj,"identifier" , &error_abort); pbus =PLATFORM_BUS_DEVICE (vms->platform_bus_dev); sbdev =SYS_BUS_DEVICE (obj); sdev.base =platform_bus_get_mmio_addr (pbus, sbdev,0 );
@@ -637,7 +639,8 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) (ID_MAPPING_ENTRY_SIZE * smmu_mapping_count);build_append_int_noprefix (table_data, node_size,2 );/* Length */ build_append_int_noprefix (table_data,4 ,1 );/* Revision */ -build_append_int_noprefix (table_data, id++,4 );/* Identifier */ +build_append_int_noprefix (table_data, sdev->id,4 );/* Identifier */ + id++;/* advance shared counter for RC/RMR node uniqueness */ /* Number of ID mappings */ build_append_int_noprefix (table_data, smmu_mapping_count,4 );/* Reference to ID Array */
hw/arm/virt.c +12 -0
@@ -254,6 +254,9 @@ static MemMapEntry extended_memmap[] = {/* Any CXL Fixed memory windows come here */ }; +/* Counts SMMUv3 devices plugged; used to assign stable IORT identifiers */ +static uint8_t smmuv3_dev_id; +static const int a15irqmap[] = { [VIRT_UART0 ] =1 , [VIRT_RTC ] =2 ,
@@ -3830,6 +3833,15 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, OBJECT(vms->sysmem), NULL);object_property_set_link (OBJECT (dev),"secure-memory" ,OBJECT (vms->secure_sysmem),NULL ); +/* +* In build_iort(), the ITS node(id=0) precedes SMMUv3 nodes +* when present. Account for it so this SMMUv3's identifier +* is globally unique across all IORT nodes. +*/ +uint8_t its_offset = (vms->msi_controller ==VIRT_MSI_CTRL_ITS ) + ?1 :0 ; +object_property_set_uint (OBJECT (dev),"identifier" , + its_offset + smmuv3_dev_id++,NULL ); }if (object_property_get_bool (OBJECT (dev),"accel" , &error_abort)) {hwaddr db_start =0 ;
Signature · 1 line
2.43.0
PATCH v5 29/32 hw/arm/smmuv3-accel: Introduce helper to query CMDQV type
In reply to Shameer KolothumIntroduce a SMMUv3AccelCmdqvType enum and a helper to query the CMDQV implementation type associated with an accelerated SMMUv3 instance.
A subsequent patch will use this helper when generating the Tegra241 CMDQV DSDT.
--- hw/arm/smmuv3-accel.h | 10 ++++++++++ hw/arm/smmuv3-accel-stubs.c | 5 +++++ hw/arm/smmuv3-accel.c | 12 ++++++++++++ hw/arm/tegra241-cmdqv.c | 6 ++++++ 4 files changed, 33 insertions(+)
hw/arm/smmuv3-accel.h +10 -0
@@ -16,6 +16,11 @@#include <linux/iommufd.h> #endif +typedef enum SMMUv3AccelCmdqvType { +SMMUV3_CMDQV_NONE =0 , +SMMUV3_CMDQV_TEGRA241 , +}SMMUv3AccelCmdqvType ; +/* * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to * support multiple VCMDQs with virtualization capabilities.
@@ -43,6 +48,10 @@ typedef struct SMMUv3AccelCmdqvOps {* If NULL, the viommu_id is freed directly via iommufd_backend_free_id(). */ void (*free_viommu)(SMMUv3State *s); +/** +* @get_type: Optional callback. Return the CMDQV implementation type. +*/ +SMMUv3AccelCmdqvType (*get_type)(void);/** * @reset: Optional callback. Reset CMDQV state. */
@@ -91,5 +100,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);bool smmuv3_accel_event_read_validate (IOMMUFDVeventq *veventq,uint32_t type,void *buf,size_t size,Error **errp);void smmuv3_accel_reset (SMMUv3State *s); +SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type (Object *obj);#endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +5 -0
@@ -57,3 +57,8 @@ bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,void smmuv3_accel_reset (SMMUv3State *s) { } + +SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type (Object *obj) +{ +return SMMUV3_CMDQV_NONE ; +}
hw/arm/smmuv3-accel.c +12 -0
@@ -1062,6 +1062,18 @@ static void smmuv3_accel_as_init(SMMUv3State *s)address_space_init (shared_as_sysmem, &root,"smmuv3-accel-as-sysmem" ); } +SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type (Object *obj) +{ +SMMUv3State *s =ARM_SMMUV3 (obj); +SMMUv3AccelState *accel = s->s_accel; + +if (!accel || !accel->cmdqv_ops || !accel->cmdqv_ops->get_type) { +return SMMUV3_CMDQV_NONE ; + } + +return accel->cmdqv_ops->get_type (); +} +static void smmuv3_machine_done (Notifier *notifier,void *data) {SMMUv3State *s =container_of (notifier, SMMUv3State, machine_done);
hw/arm/tegra241-cmdqv.c +6 -0
@@ -952,6 +952,11 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)return true; } +static SMMUv3AccelCmdqvType tegra241_cmdqv_get_type (void ) +{ +return SMMUV3_CMDQV_TEGRA241 ; +} +static bool tegra241_cmdqv_probe (SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp) {
@@ -992,6 +997,7 @@ static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = { .init = tegra241_cmdqv_init, .alloc_viommu = tegra241_cmdqv_alloc_viommu, .free_viommu = tegra241_cmdqv_free_viommu, + .get_type = tegra241_cmdqv_get_type, .reset = tegra241_cmdqv_reset, };
Signature · 1 line
2.43.0
PATCH v5 30/32 hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv.
For each accelerated SMMUv3 instance, add a Tegra241 CMDQV device object under the DSDT \_SB namespace, with HID "NVDA200C" and a UID that matches the Identifier of the corresponding SMMUv3 IORT node, so the guest OS can associate the DSDT device with the right SMMU. The _CRS covers the CMDQV MMIO aperture plus its interrupt, and _CCA declares I/O cache coherency.
See ACPI Specification 6.5, Section 6 (Device Configuration) for _HID/_UID/_CCA/_CRS.
Generated DSDT entry for a CMDQV instance paired with SMMUv3 Identifier=1:
...
Device (CV01)
{
Name (_HID, "NVDA200C") // _HID: Hardware ID
Name (_UID, One) // _UID: Unique ID
Name (_CCA, One) // _CCA: Cache Coherency Attribute
Name (_CRS, ResourceTemplate () // _CRS: Current Resource Settings
{
QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite,
0x0000000000000000, // Granularity
0x000000000C080000, // Range Minimum
0x000000000C0CFFFF, // Range Maximum
0x0000000000000000, // Translation Offset
0x0000000000050000, // Length
,, , AddressRangeMemory, TypeStatic)
Interrupt (ResourceConsumer, Edge, ActiveHigh, Exclusive, ,, )
{
0x00000094,
}
}) }
...
Generated IORT SMMUv3 node (Identifier = 1):
...
[048h 0072 001h] Type : 04
[049h 0073 002h] Length : 0058
[04Bh 0075 001h] Revision : 04
[04Ch 0076 004h] Identifier : 00000001
[050h 0080 004h] Mapping Count : 00000001
[054h 0084 004h] Mapping Offset : 00000044
...
--- hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 53 insertions(+)
hw/arm/virt-acpi-build.c +52 -0
@@ -65,6 +65,9 @@#include "target/arm/cpu.h" #include "target/arm/multiprocessing.h" +#include "smmuv3-accel.h" +#include "tegra241-cmdqv.h" +#define ARM_SPI_BASE 32#define ACPI_BUILD_TABLE_SIZE 0x20000
@@ -1121,6 +1124,51 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,build_fadt (table_data, linker, &fadt, vms->oem_id, vms->oem_table_id); } +static void acpi_dsdt_add_tegra241_cmdqv (Aml *scope,VirtMachineState *vms) +{ +for (int i =0 ; i < vms->smmuv3_devices->len; i++) { +Object *obj =OBJECT (g_ptr_array_index (vms->smmuv3_devices, i)); +PlatformBusDevice *pbus; +Aml *dev, *crs, *addr; +SysBusDevice *sbdev; +hwaddr base; +uint32_t id; +int irq; + +if (smmuv3_accel_cmdqv_type (obj) !=SMMUV3_CMDQV_TEGRA241 ) { +continue ; + } + id =object_property_get_uint (obj,"identifier" , &error_abort); + pbus =PLATFORM_BUS_DEVICE (vms->platform_bus_dev); + sbdev =SYS_BUS_DEVICE (obj); + base =platform_bus_get_mmio_addr (pbus, sbdev,1 ); + base += vms->memmap[VIRT_PLATFORM_BUS ].base; + irq =platform_bus_get_irqn (pbus, sbdev,NUM_SMMU_IRQS ); + irq += vms->irqmap[VIRT_PLATFORM_BUS ]; + irq +=ARM_SPI_BASE ; + + dev =aml_device ("CV%.02u" , id); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("NVDA200C" ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (id))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 ))); + + crs =aml_resource_template (); + addr =aml_qword_memory (AML_POS_DECODE ,AML_MIN_FIXED ,AML_MAX_FIXED , +AML_CACHEABLE ,AML_READ_WRITE ,0x0 , base, + base +TEGRA241_CMDQV_IO_LEN -0x1 ,0x0 , +TEGRA241_CMDQV_IO_LEN ); +aml_append (crs, addr); +aml_append (crs,aml_interrupt (AML_CONSUMER ,AML_EDGE , +AML_ACTIVE_HIGH ,AML_EXCLUSIVE , + (uint32_t *)&irq,1 )); +aml_append (dev,aml_name_decl ("_CRS" , crs)); + +aml_append (scope, dev); + +trace_virt_acpi_dsdt_tegra241_cmdqv (id, base, irq); + } +} +/* DSDT */ static void build_dsdt (GArray *table_data,BIOSLinker *linker,VirtMachineState *vms)
@@ -1185,6 +1233,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)acpi_dsdt_add_tpm (scope, vms);#endif +if (!vms->legacy_smmuv3_present) { +acpi_dsdt_add_tegra241_cmdqv (scope, vms); + } +aml_append (dsdt, scope); pci0_scope =aml_scope ("\\_SB.PCI0" );
hw/arm/trace-events +1 -0
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s" # virt-acpi-build.c virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out." +virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d" # smmu-common.c smmu_add_mr(const char *name) "%s"
Signature · 1 line
2.43.0
Re: [PATCH v5 30/32] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT
Quoted 49 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv.
For each accelerated SMMUv3 instance, add a Tegra241 CMDQV device object under the DSDT \_SB namespace, with HID "NVDA200C" and a UID that matches the Identifier of the corresponding SMMUv3 IORT node, so the guest OS can associate the DSDT device with the right SMMU. The
_CRScovers the CMDQV MMIO aperture plus its interrupt, and_CCAdeclares I/O cache coherency.See ACPI Specification 6.5, Section 6 (Device Configuration) for _HID/_UID/_CCA/_CRS.
Generated DSDT entry for a CMDQV instance paired with SMMUv3 Identifier=1:
...
Device (CV01){ Name (_HID, "NVDA200C") // _HID: Hardware ID Name (_UID, One) // _UID: Unique ID Name (_CCA, One) // _CCA: Cache Coherency Attribute Name (_CRS, ResourceTemplate () // _CRS: Current Resource Settings { QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite, 0x0000000000000000, // Granularity 0x000000000C080000, // Range Minimum 0x000000000C0CFFFF, // Range Maximum 0x0000000000000000, // Translation Offset 0x0000000000050000, // Length ,, , AddressRangeMemory, TypeStatic) Interrupt (ResourceConsumer, Edge, ActiveHigh, Exclusive, ,, ) { 0x00000094, } })}
...
Generated IORT SMMUv3 node (Identifier = 1):...
[048h 0072 001h] Type : 04
[049h 0073 002h] Length : 0058
[04Bh 0075 001h] Revision : 04
[04Ch 0076 004h] Identifier : 00000001
[050h 0080 004h] Mapping Count : 00000001
[054h 0084 004h] Mapping Offset : 00000044
...
Quoted 3 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Eric
Quoted 94 lines
--- hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 53 insertions(+)hw/arm/virt-acpi-build.c +52
-0@@ -65,6 +65,9 @@#include "target/arm/cpu.h" #include "target/arm/multiprocessing.h" +#include "smmuv3-accel.h" +#include "tegra241-cmdqv.h" +#define ARM_SPI_BASE 32#define ACPI_BUILD_TABLE_SIZE 0x20000@@ -1121,6 +1124,51 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,build_fadt (table_data, linker, &fadt, vms->oem_id, vms->oem_table_id); } +static void acpi_dsdt_add_tegra241_cmdqv (Aml *scope,VirtMachineState *vms) +{ +for (int i =0 ; i < vms->smmuv3_devices->len; i++) { +Object *obj =OBJECT (g_ptr_array_index (vms->smmuv3_devices, i)); +PlatformBusDevice *pbus; +Aml *dev, *crs, *addr; +SysBusDevice *sbdev; +hwaddr base; +uint32_t id; +int irq; + +if (smmuv3_accel_cmdqv_type (obj) !=SMMUV3_CMDQV_TEGRA241 ) { +continue ; + } + id =object_property_get_uint (obj,"identifier" , &error_abort); + pbus =PLATFORM_BUS_DEVICE (vms->platform_bus_dev); + sbdev =SYS_BUS_DEVICE (obj); + base =platform_bus_get_mmio_addr (pbus, sbdev,1 ); + base += vms->memmap[VIRT_PLATFORM_BUS ].base; + irq =platform_bus_get_irqn (pbus, sbdev,NUM_SMMU_IRQS ); + irq += vms->irqmap[VIRT_PLATFORM_BUS ]; + irq +=ARM_SPI_BASE ; + + dev =aml_device ("CV%.02u" , id); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("NVDA200C" ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (id))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 ))); + + crs =aml_resource_template (); + addr =aml_qword_memory (AML_POS_DECODE ,AML_MIN_FIXED ,AML_MAX_FIXED , +AML_CACHEABLE ,AML_READ_WRITE ,0x0 , base, + base +TEGRA241_CMDQV_IO_LEN -0x1 ,0x0 , +TEGRA241_CMDQV_IO_LEN ); +aml_append (crs, addr); +aml_append (crs,aml_interrupt (AML_CONSUMER ,AML_EDGE , +AML_ACTIVE_HIGH ,AML_EXCLUSIVE , + (uint32_t *)&irq,1 )); +aml_append (dev,aml_name_decl ("_CRS" , crs)); + +aml_append (scope, dev); + +trace_virt_acpi_dsdt_tegra241_cmdqv (id, base, irq); + } +} +/* DSDT */ static void build_dsdt (GArray *table_data,BIOSLinker *linker,VirtMachineState *vms)@@ -1185,6 +1233,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)acpi_dsdt_add_tpm (scope, vms);#endif +if (!vms->legacy_smmuv3_present) { +acpi_dsdt_add_tegra241_cmdqv (scope, vms); + } +aml_append (dsdt, scope); pci0_scope =aml_scope ("\\_SB.PCI0" );hw/arm/trace-events +1
-0@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s" # virt-acpi-build.c virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out." +virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d" # smmu-common.c smmu_add_mr(const char *name) "%s"
PATCH v5 31/32 hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active
In reply to Shameer KolothumWhen CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association. Block its hot-unplug to preserve this association and the guest's boot time CMDQV configuration.
Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.
--- hw/arm/smmuv3-accel.h | 1 + hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
hw/arm/smmuv3-accel.h +1 -0
@@ -85,6 +85,7 @@ typedef struct SMMUv3AccelDevice {IOMMUFDVdev *vdev;QLIST_ENTRY (SMMUv3AccelDevice) next;SMMUv3AccelState *s_accel; +Error *unplug_blocker;/* set when CMDQV is active to block hot-unplug */ } SMMUv3AccelDevice;bool smmuv3_accel_init (SMMUv3State *s,Error **errp);
hw/arm/smmuv3-accel.c +18 -0
@@ -761,6 +761,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,return false; } +/* +* CMDQV is active: block hot-unplug of the device that established the +* viommu association. Removing it would cause the vIOMMU to host SMMUv3 +* association be changed via device hot-plug. +*/ +if (s->s_accel->cmdqv_ops) { +PCIDevice *pdev =pci_find_device (bus,pci_bus_num (bus), devfn); +error_setg (&accel_dev->unplug_blocker, +"CMDQV is active: removing the device that established the " +"viommu association would break the guest CMDQV" ); +qdev_add_unplug_blocker (DEVICE (pdev), accel_dev->unplug_blocker); + } done: accel_dev->hiodi = hiodi; accel_dev->s_accel = s->s_accel;
@@ -1088,6 +1100,12 @@ static void smmuv3_machine_done(Notifier *notifier, void *data) "at least one cold-plugged VFIO device");exit (1 ); } + +if (s->cmdqv ==ON_OFF_AUTO_ON && !accel->cmdqv) { +error_report ("arm-smmuv3 cmdqv=on requires at least one cold-plugged " +"VFIO device" ); +exit (1 ); + } }bool smmuv3_accel_init (SMMUv3State *s,Error **errp)
Signature · 1 line
2.43.0
Re: [PATCH v5 31/32] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active
Quoted 6 lines · Shameer Kolothum
When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association. Block its hot-unplug to preserve this association and the guest's boot time CMDQV configuration.
Also abort at
machine_doneif cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.
To me this is still missing the "why", something along what you put in
https://lore.kernel.org/all/IA0PR12MB755515924AD70A273B7879A1AB3D2@IA0PR12MB7555.namprd12.prod.outlook.com/
Thanks
Eric
Quoted 56 lines
Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/smmuv3-accel.h | 1 + hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)hw/arm/smmuv3-accel.h +1
-0@@ -85,6 +85,7 @@ typedef struct SMMUv3AccelDevice {IOMMUFDVdev *vdev;QLIST_ENTRY (SMMUv3AccelDevice) next;SMMUv3AccelState *s_accel; +Error *unplug_blocker;/* set when CMDQV is active to block hot-unplug */ } SMMUv3AccelDevice;bool smmuv3_accel_init (SMMUv3State *s,Error **errp);hw/arm/smmuv3-accel.c +18
-0@@ -761,6 +761,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,return false; } +/* +* CMDQV is active: block hot-unplug of the device that established the +* viommu association. Removing it would cause the vIOMMU to host SMMUv3 +* association be changed via device hot-plug. +*/ +if (s->s_accel->cmdqv_ops) { +PCIDevice *pdev =pci_find_device (bus,pci_bus_num (bus), devfn); +error_setg (&accel_dev->unplug_blocker, +"CMDQV is active: removing the device that established the " +"viommu association would break the guest CMDQV" ); +qdev_add_unplug_blocker (DEVICE (pdev), accel_dev->unplug_blocker); + } done: accel_dev->hiodi = hiodi; accel_dev->s_accel = s->s_accel;@@ -1088,6 +1100,12 @@ static void smmuv3_machine_done(Notifier *notifier, void *data) "at least one cold-plugged VFIO device");exit (1 ); } + +if (s->cmdqv ==ON_OFF_AUTO_ON && !accel->cmdqv) { +error_report ("arm-smmuv3 cmdqv=on requires at least one cold-plugged " +"VFIO device" ); +exit (1 ); + } }bool smmuv3_accel_init (SMMUv3State *s,Error **errp)
RE: [PATCH v5 31/32] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active
Quoted 23 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:09
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 31/32] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is activeExternal email: Use caution opening links or attachments
Quoted 4 lines · Shameer Kolothum
When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association. Block its hot-unplug to preserve this association and the guest's boot time CMDQV configuration.
Quoted 6 lines
Quoted 2 lines
Also abort at
machine_doneif cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.To me this is still missing the "why", something along what you put in
https://lore.kernel.org/all/IA0PR12MB755515924AD70A273B7879A1AB3D
[email protected]/
Ok. I will summarise that and add into this commit msg.
Thanks,
Shameer
PATCH v5 32/32 hw/arm/smmuv3: Add cmdqv property for SMMUv3 device
In reply to Shameer KolothumIntroduce a "cmdqv" property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.
--- hw/arm/smmuv3.c | 8 ++++++++ qemu-options.hx | 8 ++++++++ 2 files changed, 16 insertions(+)
hw/arm/smmuv3.c +8 -0
@@ -1994,6 +1994,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp) "bits if accel=on");return false; } +if (s->cmdqv ==ON_OFF_AUTO_ON ) { +error_setg (errp,"cmdqv can only be enabled if accel=on" ); +return false; + }return true; }
@@ -2144,6 +2148,7 @@ static const Property smmuv3_properties[] = {DEFINE_PROP_OAS_MODE ("oas" , SMMUv3State, oas,OAS_MODE_AUTO ),DEFINE_PROP_SSIDSIZE_MODE ("ssidsize" , SMMUv3State, ssidsize,SSID_SIZE_MODE_AUTO ), + DEFINE_PROP_ON_OFF_AUTO("cmdqv", SMMUv3State, cmdqv, ON_OFF_AUTO_AUTO), };static void smmuv3_instance_init (Object *obj)
@@ -2188,6 +2193,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"A value of N allows SSIDs in the range [0 .. 2^N - 1]. " "A value of 0 disables SubstreamID support. A value greater " "than 0 is required to enable PASID support."); +object_class_property_set_description (klass,"cmdqv" , +"Enable/disable CMDQV support (for accel=on). " +"Valid values are on, off, and auto. Defaults to auto." ); } static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
qemu-options.hx +8 -0
@@ -1329,6 +1329,14 @@ SRST - With accel=on, auto means the value is automatically derived from the host SMMU. - With accel=off, auto is resolved to 0. + ``cmdqv=on|off|auto`` (default: auto) + Enable hardware Command Queue Virtualization (CMDQV) for the + SMMUv3 command queue. Currently only the NVIDIA Tegra241 CMDQV + implementation is supported. + + - With accel=on, auto means the value is automatically derived from the host SMMU. + - With accel=off, auto is resolved to 'off'. + ``-device amd-iommu[,option=...]`` Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU). Only available with ``-machine q35``, it supports the following options:
Signature · 1 line
2.43.0
Re: [PATCH v5 32/32] hw/arm/smmuv3: Add cmdqv property for SMMUv3 device
Quoted 5 lines · Shameer Kolothum
Introduce a "cmdqv" property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.
Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Thanks
Eric
Quoted 57 lines
--- hw/arm/smmuv3.c | 8 ++++++++ qemu-options.hx | 8 ++++++++ 2 files changed, 16 insertions(+)hw/arm/smmuv3.c +8
-0@@ -1994,6 +1994,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp) "bits if accel=on");return false; } +if (s->cmdqv ==ON_OFF_AUTO_ON ) { +error_setg (errp,"cmdqv can only be enabled if accel=on" ); +return false; + }return true; }@@ -2144,6 +2148,7 @@ static const Property smmuv3_properties[] = {DEFINE_PROP_OAS_MODE ("oas" , SMMUv3State, oas,OAS_MODE_AUTO ),DEFINE_PROP_SSIDSIZE_MODE ("ssidsize" , SMMUv3State, ssidsize,SSID_SIZE_MODE_AUTO ), + DEFINE_PROP_ON_OFF_AUTO("cmdqv", SMMUv3State, cmdqv, ON_OFF_AUTO_AUTO), };static void smmuv3_instance_init (Object *obj)@@ -2188,6 +2193,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"A value of N allows SSIDs in the range [0 .. 2^N - 1]. " "A value of 0 disables SubstreamID support. A value greater " "than 0 is required to enable PASID support."); +object_class_property_set_description (klass,"cmdqv" , +"Enable/disable CMDQV support (for accel=on). " +"Valid values are on, off, and auto. Defaults to auto." ); } static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,qemu-options.hx +8
-0@@ -1329,6 +1329,14 @@ SRST - With accel=on, auto means the value is automatically derived from the host SMMU. - With accel=off, auto is resolved to 0. + ``cmdqv=on|off|auto`` (default: auto) + Enable hardware Command Queue Virtualization (CMDQV) for the + SMMUv3 command queue. Currently only the NVIDIA Tegra241 CMDQV + implementation is supported. + + - With accel=on, auto means the value is automatically derived from the host SMMU. + - With accel=off, auto is resolved to 'off'. + ``-device amd-iommu[,option=...]`` Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU). Only available with ``-machine q35``, it supports the following options: