hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3
Subsystems (2)
v6 cover hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3
Hi,
Changes from v5:
https://lore.kernel.org/qemu-devel/[email protected]/
- Addressed v5 feedback and picked up Reviewed-by tags. Thanks!
- Added documentation for design and lifecycle details. Please see patch #30.
- Changed probe() and alloc_viommu() CMDQV ops both mandatory (patches #6, #8)
- Fixed build on windows failure (patch #11)
- Improved tracing with "Direct" vs "VI" info (patches #15, #16)
- Removed vcmdq_enabled() check on BASE reprogramming (patch #16)
- Dropped address_space_range_is_ram() check; record error on setup_vcmdq() failure (patch #18)
- Dropped qemu_ram_backend_pagesize_min() and instead uses qemu_minrampagesize()- patch #25
- Based on top of Nathan's "Resolve AUTO properties" v4 series [0].
Please find the complete branch here:
https://github.com/shamiali2008/qemu-master/tree/master-vcmdq-v6-ext
Sanity tested on NVIDIA Grace. Further testing in progress.
Feedback and testing are very welcome.
Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/
---
Background(from RFCv1):
https://lore.kernel.org/qemu-devel/[email protected]/
Thanks to Nicolin for the initial patches and testing on which this is based.
Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.
Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)
Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.
VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].
---
Nicolin Chen (15):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Implement CMDQV init hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT
Shameer Kolothum (16):
system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq hw/arm/smmuv3-accel: Introduce CMDQV ops interface hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle hw/arm/virt: Use stored SMMUv3 device list for IORT build hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
memory: Allow RAM device regions to skip IOMMU mapping hw/arm/smmuv3-accel: Introduce common helper for veventq read hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors hw/arm/smmuv3: Add per-device identifier property hw/arm/smmuv3-accel: Introduce helper to query CMDQV type hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle hw/arm/smmuv3: Add cmdqv property for SMMUv3 device
hw/arm/smmuv3-accel.h | 48 ++ hw/arm/tegra241-cmdqv.h | 384 +++++++++++ include/hw/arm/smmuv3.h | 3 + include/hw/arm/virt.h | 1 + include/system/iommufd.h | 17 +- include/system/memory.h | 21 + backends/iommufd.c | 64 ++ hw/arm/smmuv3-accel-stubs.c | 12 + hw/arm/smmuv3-accel.c | 194 ++++-- hw/arm/smmuv3.c | 10 + hw/arm/tegra241-cmdqv-stubs.c | 16 + hw/arm/tegra241-cmdqv.c | 1122 +++++++++++++++++++++++++++++++++ hw/arm/virt-acpi-build.c | 127 ++-- hw/arm/virt.c | 37 ++ hw/vfio/iommufd.c | 4 +- hw/vfio/listener.c | 6 + system/memory.c | 10 + backends/trace-events | 4 +- hw/arm/Kconfig | 5 + hw/arm/meson.build | 2 + hw/arm/trace-events | 11 + hw/vfio/trace-events | 1 + qemu-options.hx | 8 + 23 files changed, 2027 insertions(+), 80 deletions(-) create mode 100644 hw/arm/tegra241-cmdqv.h create mode 100644 hw/arm/tegra241-cmdqv-stubs.c create mode 100644 hw/arm/tegra241-cmdqv.c
Signature · 1 line
2.43.0
PATCH v6 1/31 backends/iommufd: Update iommufd_backend_get_device_info
From: Nicolin Chen <[email protected]>
The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass a type value explicitly initialised to zero (IOMMU_HW_INFO_TYPE_DEFAULT) when no specific type is requested.
--- backends/iommufd.c | 7 +++++++ hw/arm/smmuv3-accel.c | 2 +- hw/vfio/iommufd.c | 4 ++-- 3 files changed, 10 insertions(+), 3 deletions(-)
backends/iommufd.c +7 -0
@@ -390,16 +390,23 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be,return true; } +/* +* @type can carry a desired HW info type defined in the uapi headers. If caller +* doesn't have one, indicating it wants the default type, then @type should be +* zeroed (i.e. IOMMU_HW_INFO_TYPE_DEFAULT). +*/ bool iommufd_backend_get_device_info (IOMMUFDBackend *be,uint32_t devid,uint32_t *type,void *data,uint32_t len,uint64_t *caps,uint8_t *max_pasid_log2,Error **errp) {struct iommu_hw_info info = { + .flags = (*type) ?IOMMU_HW_INFO_FLAG_INPUT_TYPE :0 , .size =sizeof (info), .dev_id = devid, .data_len = len, .data_uptr = (uintptr_t )data, + .in_data_type = *type, };if (ioctl (be->fd,IOMMU_GET_HW_INFO , &info)) {
hw/arm/smmuv3-accel.c +1 -1
@@ -176,7 +176,7 @@ smmuv3_accel_hw_compatible(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, Error **errp) { struct iommu_hw_info_arm_smmuv3 info; - uint32_t data_type; + uint32_t data_type = IOMMU_HW_INFO_TYPE_DEFAULT; uint64_t caps; if (!iommufd_backend_get_device_info(hiodi->iommufd, hiodi->devid,
hw/vfio/iommufd.c +2 -2
@@ -353,7 +353,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,IOMMUFDBackend *iommufd = vbasedev->iommufd;VFIOContainer *bcontainer =VFIO_IOMMU (container);bool viommu_nesting, viommu_nesting_dirty; -uint32_t type, flags =0 ; +uint32_t type =IOMMU_HW_INFO_TYPE_DEFAULT , flags =0 ;uint64_t hw_caps;VendorCaps caps;VFIOIOASHwpt *hwpt;
@@ -948,7 +948,7 @@ static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque,HostIOMMUDeviceIOMMUFD *hiodi;HostIOMMUDeviceCaps *caps = &hiod->caps;VendorCaps *vendor_caps = &caps->vendor_caps; -enum iommu_hw_info_type type; +uint32_t type =IOMMU_HW_INFO_TYPE_DEFAULT ;uint8_t max_pasid_log2;uint64_t hw_caps;
Signature · 1 line
2.43.0
PATCH v6 2/31 backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
The updated IOMMUFD VIOMMU_ALLOC uAPI allows userspace to provide a data buffer when creating a vIOMMU (e.g. for Tegra241 CMDQV). Extend iommufd_backend_alloc_viommu() to pass a user pointer and size to the kernel.
Update the caller accordingly.
--- include/system/iommufd.h | 1 + backends/iommufd.c | 4 ++++ hw/arm/smmuv3-accel.c | 4 ++-- backends/trace-events | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-)
include/system/iommufd.h +1 -0
@@ -89,6 +89,7 @@ bool iommufd_backend_alloc_hwpt(IOMMUFDBackend *be, uint32_t dev_id, Error **errp);bool iommufd_backend_alloc_viommu (IOMMUFDBackend *be,uint32_t dev_id,uint32_t viommu_type,uint32_t hwpt_id, +void *data_ptr,uint32_t data_len,uint32_t *out_hwpt,Error **errp); bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
backends/iommufd.c +4 -0
@@ -463,6 +463,7 @@ bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id,bool iommufd_backend_alloc_viommu (IOMMUFDBackend *be,uint32_t dev_id,uint32_t viommu_type,uint32_t hwpt_id, +void *data_ptr,uint32_t data_len,uint32_t *out_viommu_id,Error **errp) {int ret;
@@ -471,11 +472,14 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id, .type = viommu_type, .dev_id = dev_id, .hwpt_id = hwpt_id, + .data_len = data_len, + .data_uptr = (uintptr_t)data_ptr, }; ret =ioctl (be->fd,IOMMU_VIOMMU_ALLOC , &alloc_viommu);trace_iommufd_backend_alloc_viommu (be->fd, dev_id, viommu_type, hwpt_id, + (uintptr_t )data_ptr, data_len, alloc_viommu.out_viommu_id, ret);if (ret) {error_setg_errno (errp, errno,"IOMMU_VIOMMU_ALLOC failed" );
hw/arm/smmuv3-accel.c +2 -2
@@ -582,8 +582,8 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,IOMMUFDViommu *viommu;if (!iommufd_backend_alloc_viommu (hiodi->iommufd, hiodi->devid, -IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , - s2_hwpt_id, &viommu_id, errp)) { +IOMMU_VIOMMU_TYPE_ARM_SMMUV3 , s2_hwpt_id, +NULL ,0 , &viommu_id, errp)) {return false; }
backends/trace-events +1 -1
@@ -21,7 +21,7 @@ iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (% iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%u enable=%d (%d)" iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, uint64_t size, uint64_t flags, uint64_t page_size, int ret) " iommufd=%d hwpt=%u iova=0x%"PRIx64" size=0x%"PRIx64" flags=0x%"PRIx64" page_size=0x%"PRIx64" (%d)" iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, uint32_t entry_len, uint32_t entry_num, uint32_t done_num, uint64_t data_ptr, int ret) " iommufd=%d id=%u data_type=%u entry_len=%u entry_num=%u done_num=%u data_ptr=0x%"PRIx64" (%d)" -iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)" +iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)" iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)" iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
Signature · 1 line
2.43.0
PATCH v6 3/31 backends/iommufd: Introduce iommufd_backend_alloc_hw_queue
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add a helper to allocate an iommufd backed HW queue for a vIOMMU.
While at it, define a struct IOMMUFDHWqueue for use by vendor implementations.
--- include/system/iommufd.h | 11 +++++++++++ backends/iommufd.c | 31 +++++++++++++++++++++++++++++++ backends/trace-events | 1 + 3 files changed, 43 insertions(+)
include/system/iommufd.h +11 -0
@@ -65,6 +65,12 @@ typedef struct IOMMUFDVeventq {bool event_start;/* True after first valid event; cleared on overflow */ } IOMMUFDVeventq; +/* HW queue object for a vIOMMU-specific HW-accelerated queue */ +typedef struct IOMMUFDHWqueue { +IOMMUFDViommu *viommu; +uint32_t hw_queue_id; +}IOMMUFDHWqueue ; +bool iommufd_backend_connect (IOMMUFDBackend *be,Error **errp);void iommufd_backend_disconnect (IOMMUFDBackend *be);
@@ -101,6 +107,11 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,uint32_t *out_veventq_id, uint32_t *out_veventq_fd, Error **errp); +bool iommufd_backend_alloc_hw_queue (IOMMUFDBackend *be,uint32_t viommu_id, +uint32_t queue_type,uint32_t index, +uint64_t addr,uint64_t length, +uint32_t *out_hw_queue_id,Error **errp); +bool iommufd_backend_set_dirty_tracking (IOMMUFDBackend *be,uint32_t hwpt_id,bool start,Error **errp); bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +31 -0
@@ -549,6 +549,37 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,return true; } +bool iommufd_backend_alloc_hw_queue (IOMMUFDBackend *be,uint32_t viommu_id, +uint32_t queue_type,uint32_t index, +uint64_t addr,uint64_t length, +uint32_t *out_hw_queue_id,Error **errp) +{ +int ret; +struct iommu_hw_queue_alloc alloc_hw_queue = { + .size =sizeof (alloc_hw_queue), + .flags =0 , + .viommu_id = viommu_id, + .type = queue_type, + .index = index, + .nesting_parent_iova = addr, + .length = length, + }; + + ret =ioctl (be->fd,IOMMU_HW_QUEUE_ALLOC , &alloc_hw_queue); + +trace_iommufd_backend_alloc_hw_queue (be->fd, viommu_id, queue_type, + index, addr, length, + alloc_hw_queue.out_hw_queue_id, ret); +if (ret) { +error_setg_errno (errp, errno,"IOMMU_HW_QUEUE_ALLOC failed" ); +return false; + } + +g_assert (out_hw_queue_id); + *out_hw_queue_id = alloc_hw_queue.out_hw_queue_id; +return true; +} +bool host_iommu_device_iommufd_attach_hwpt (HostIOMMUDeviceIOMMUFD *hiodi,uint32_t hwpt_id,Error **errp) {
backends/trace-events +1 -0
@@ -24,6 +24,7 @@ iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)" iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)" iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)" +iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)" # igvm-cfg.c igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
PATCH v6 4/31 backends/iommufd: Introduce iommufd_backend_viommu_mmap
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add a backend helper to mmap hardware MMIO regions exposed via iommufd for a vIOMMU instance. This allows user space to access HW-accelerated MMIO pages provided by the vIOMMU.
The caller is responsible for unmapping the returned region.
--- include/system/iommufd.h | 4 ++++ backends/iommufd.c | 22 ++++++++++++++++++++++ backends/trace-events | 1 + 3 files changed, 27 insertions(+)
include/system/iommufd.h +4 -0
@@ -112,6 +112,10 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id, uint64_t addr, uint64_t length, uint32_t *out_hw_queue_id, Error **errp); +bool iommufd_backend_viommu_mmap (IOMMUFDBackend *be,uint32_t viommu_id, +uint64_t size,off_t offset,void **out_ptr, +Error **errp); +bool iommufd_backend_set_dirty_tracking (IOMMUFDBackend *be,uint32_t hwpt_id,bool start,Error **errp); bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +22 -0
@@ -580,6 +580,28 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,return true; } +/* +* Helper to mmap HW MMIO regions exposed via iommufd for a vIOMMU instance. +* The caller is responsible for unmapping the mapped region. +*/ +bool iommufd_backend_viommu_mmap (IOMMUFDBackend *be,uint32_t viommu_id, +uint64_t size,off_t offset,void **out_ptr, +Error **errp) +{ +g_assert (viommu_id); +g_assert (out_ptr); + + *out_ptr =mmap (NULL , size,PROT_READ |PROT_WRITE ,MAP_SHARED , be->fd, + offset); +trace_iommufd_backend_viommu_mmap (be->fd, viommu_id, size, offset); +if (*out_ptr ==MAP_FAILED ) { +error_setg_errno (errp, errno,"IOMMUFD vIOMMU mmap failed" ); +return false; + } + +return true; +} +bool host_iommu_device_iommufd_attach_hwpt (HostIOMMUDeviceIOMMUFD *hiodi,uint32_t hwpt_id,Error **errp) {
backends/trace-events +1 -0
@@ -25,6 +25,7 @@ iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)" iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)" iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)" +iommufd_backend_viommu_mmap(int iommufd, uint32_t viommu_id, uint64_t size, uint64_t offset) " iommufd=%d viommu_id=%u size=0x%"PRIx64" offset=0x%"PRIx64 # igvm-cfg.c igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
PATCH v6 5/31 system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq
In reply to Shameer KolothumThe viommu field is assigned but never used. Callers freeing the veventq already have access to the IOMMUFDViommu object through other references, so this field is redundant.
Removing it also simplifies upcoming changes where veventq is allocated based on the viommu id before the IOMMUFDViommu object is created (e.g. vendor CMDQV-based veventq allocation).
No functional change.
--- include/system/iommufd.h | 1 - hw/arm/smmuv3-accel.c | 1 - 2 files changed, 2 deletions(-)
include/system/iommufd.h +0 -1
@@ -58,7 +58,6 @@ typedef struct IOMMUFDVdev {/* Virtual event queue interface for a vIOMMU */ typedef struct IOMMUFDVeventq { - IOMMUFDViommu *viommu; uint32_t veventq_id; uint32_t veventq_fd; uint32_t last_event_seq; /* Sequence number of last processed event */
hw/arm/smmuv3-accel.c +0 -1
@@ -553,7 +553,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp) veventq =g_new0 (IOMMUFDVeventq,1 ); veventq->veventq_id = veventq_id; veventq->veventq_fd = veventq_fd; - veventq->viommu = accel->viommu; accel->veventq = veventq;/* Set up event handler for veventq fd */
Signature · 1 line
2.43.0
PATCH v6 6/31 hw/arm/smmuv3-accel: Introduce CMDQV ops interface
In reply to Shameer KolothumCommand Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.
For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).
The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.
CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.
This provides an extensible design to support future vendor-specific CMDQV implementations.
No functional change.
--- hw/arm/smmuv3-accel.h | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
hw/arm/smmuv3-accel.h +34 -0
@@ -10,11 +10,44 @@#define HW_ARM_SMMUV3_ACCEL_H #include "hw/arm/smmu-common.h" +#include "hw/arm/smmuv3.h" #include "system/iommufd.h" #ifdef CONFIG_LINUX #include <linux/iommufd.h> #endif +/* +* CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to +* support multiple VCMDQs with virtualization capabilities. +* CMDQV specific behavior is factored behind this ops interface. +*/ +typedef struct SMMUv3AccelCmdqvOps { +/** +* @probe: Mandatory. Vendor-specific device probing. +*/ +bool (*probe)(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp); +/** +* @init: Optional callback. Initialize CMDQV hardware. +*/ +bool (*init)(SMMUv3State *s,Error **errp); +/** +* @alloc_viommu: Mandatory. Allocate CMDQV viommu resources. +*/ +bool (*alloc_viommu)(SMMUv3State *s, +HostIOMMUDeviceIOMMUFD *idev, +uint32_t *out_viommu_id, +Error **errp); +/** +* @free_viommu: Optional callback. Free CMDQV viommu resources. +* If NULL, the viommu_id is freed directly via iommufd_backend_free_id(). +*/ +void (*free_viommu)(SMMUv3State *s); +/** +* @reset: Optional callback. Reset CMDQV state. +*/ +void (*reset)(SMMUv3State *s); +}SMMUv3AccelCmdqvOps ; +/* * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object. * Holds bypass and abort proxy HWPT IDs used for device attachment.
@@ -27,6 +60,7 @@ typedef struct SMMUv3AccelState { QLIST_HEAD(, SMMUv3AccelDevice) device_list;bool auto_mode;bool auto_finalised; +const SMMUv3AccelCmdqvOps *cmdqv_ops; } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
Signature · 1 line
2.43.0
Re: [PATCH v6 06/31] hw/arm/smmuv3-accel: Introduce CMDQV ops interface
Hi Shameer,
Quoted 16 lines · Shameer Kolothum
Command Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.
For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).
The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.
CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.
Quoted 7 lines
This provides an extensible design to support future vendor-specific CMDQV implementations.
No functional change.
Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Thanks
Eric
Quoted 61 lines
--- hw/arm/smmuv3-accel.h | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)hw/arm/smmuv3-accel.h +34
-0@@ -10,11 +10,44 @@#define HW_ARM_SMMUV3_ACCEL_H #include "hw/arm/smmu-common.h" +#include "hw/arm/smmuv3.h" #include "system/iommufd.h" #ifdef CONFIG_LINUX #include <linux/iommufd.h> #endif +/* +* CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to +* support multiple VCMDQs with virtualization capabilities. +* CMDQV specific behavior is factored behind this ops interface. +*/ +typedef struct SMMUv3AccelCmdqvOps { +/** +* @probe: Mandatory. Vendor-specific device probing. +*/ +bool (*probe)(SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp); +/** +* @init: Optional callback. Initialize CMDQV hardware. +*/ +bool (*init)(SMMUv3State *s,Error **errp); +/** +* @alloc_viommu: Mandatory. Allocate CMDQV viommu resources. +*/ +bool (*alloc_viommu)(SMMUv3State *s, +HostIOMMUDeviceIOMMUFD *idev, +uint32_t *out_viommu_id, +Error **errp); +/** +* @free_viommu: Optional callback. Free CMDQV viommu resources. +* If NULL, the viommu_id is freed directly via iommufd_backend_free_id(). +*/ +void (*free_viommu)(SMMUv3State *s); +/** +* @reset: Optional callback. Reset CMDQV state. +*/ +void (*reset)(SMMUv3State *s); +}SMMUv3AccelCmdqvOps ; +/* * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object. * Holds bypass and abort proxy HWPT IDs used for device attachment. @@ -27,6 +60,7 @@ typedef struct SMMUv3AccelState { QLIST_HEAD(, SMMUv3AccelDevice) device_list;bool auto_mode;bool auto_finalised; +const SMMUv3AccelCmdqvOps *cmdqv_ops; } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
PATCH v6 7/31 hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub
In reply to Shameer KolothumIntroduce a Tegra241 CMDQV backend that plugs into the SMMUv3 accelerated CMDQV ops interface.
This patch wires up the Tegra241 CMDQV backend and provides a stub implementation for CMDQV probe, initialization, vIOMMU allocation and reset handling.
Functional CMDQV support is added in follow-up patches.
--- hw/arm/tegra241-cmdqv.h | 15 ++++++++++ hw/arm/tegra241-cmdqv-stubs.c | 16 ++++++++++ hw/arm/tegra241-cmdqv.c | 56 +++++++++++++++++++++++++++++++++++ hw/arm/Kconfig | 5 ++++ hw/arm/meson.build | 2 ++ 5 files changed, 94 insertions(+) create mode 100644 hw/arm/tegra241-cmdqv.h create mode 100644 hw/arm/tegra241-cmdqv-stubs.c create mode 100644 hw/arm/tegra241-cmdqv.c
hw/arm/tegra241-cmdqv.h new file +15 -0
@@ -0,0 +1,15 @@ +/* +* Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved +* NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3 +* +* Written by Nicolin Chen, Shameer Kolothum +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_ARM_TEGRA241_CMDQV_H +#define HW_ARM_TEGRA241_CMDQV_H + +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void ); + +#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv-stubs.c new file +16 -0
@@ -0,0 +1,16 @@ +/* +* Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved +* +* Stubs for Tegra241 CMDQ-Virtualization extension for SMMUv3 +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "smmuv3-accel.h" +#include "hw/arm/tegra241-cmdqv.h" + +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void ) +{ +return NULL ; +}
hw/arm/tegra241-cmdqv.c new file +56 -0
56 lines not on this page · raw message
hw/arm/Kconfig +5 -0
@@ -642,6 +642,10 @@ config FSL_IMX8MM_EVK depends on TCG select FSL_IMX8MM +config TEGRA241_CMDQV + bool + depends on ARM_SMMUV3_ACCEL + config ARM_SMMUV3_ACCEL bool depends on ARM_SMMUV3
@@ -649,6 +653,7 @@ config ARM_SMMUV3_ACCEL config ARM_SMMUV3 bool select ARM_SMMUV3_ACCEL if IOMMUFD + imply TEGRA241_CMDQV config FSL_IMX6UL bool
hw/arm/meson.build +2 -0
@@ -89,6 +89,8 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MM_EVK', if_true: files('imx8mm-evk.c')) arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c')) arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c')) stub_ss.add(files('smmuv3-accel-stubs.c')) +arm_common_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c')) +stub_ss.add(files('tegra241-cmdqv-stubs.c')) arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c')) arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c')) arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
Signature · 1 line
2.43.0
PATCH v6 8/31 hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle
In reply to Shameer KolothumAdd support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.
If set to OFF, CMDQV is not used and the default IOMMUFD-backed allocation path is taken.
If set to AUTO, QEMU attempts to probe a CMDQV backend during device setup. If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.
If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.
When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.
The current implementation wires up the Tegra241 CMDQV backend through the generic ops interface. Functional CMDQV behaviour is added in subsequent patches.
No functional change.
--- include/hw/arm/smmuv3.h | 2 + hw/arm/smmuv3-accel.c | 94 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 89 insertions(+), 7 deletions(-)
include/hw/arm/smmuv3.h +2 -0
@@ -75,6 +75,8 @@ struct SMMUv3State {OnOffAuto ats;OasMode oas;SsidSizeMode ssidsize; +/* SMMU CMDQV extension */ +OnOffAuto cmdqv;Notifier machine_done; };
hw/arm/smmuv3-accel.c +87 -7
135 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v6 9/31 hw/arm/virt: Use stored SMMUv3 device list for IORT build
In reply to Shameer KolothumIntroduce a GPtrArray in VirtMachineState to track all SMMUv3 devices created on the virt machine, and use it when building the IORT table instead of relying on object_child_foreach_recursive() walks of the object tree.
This avoids recursive object traversal and provides a foundation for subsequent patches that need direct access to SMMUv3 instances for CMDQV-related handling.
No functional change. No bios-tables qtest failures observed.
--- include/hw/arm/virt.h | 1 + hw/arm/virt-acpi-build.c | 70 ++++++++++++++++++---------------------- hw/arm/virt.c | 3 ++ 3 files changed, 35 insertions(+), 39 deletions(-)
include/hw/arm/virt.h +1 -0
@@ -207,6 +207,7 @@ struct VirtMachineState {MemoryRegion *sysmem;MemoryRegion *secure_sysmem;bool pci_preserve_config; +GPtrArray *smmuv3_devices; };#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
hw/arm/virt-acpi-build.c +31 -39
87 lines not on this page · raw message
hw/arm/virt.c +3 -0
@@ -3865,6 +3865,7 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, }create_smmuv3_dev_dtb (vms, dev, bus, errp); +g_ptr_array_add (vms->smmuv3_devices, dev); } }
@@ -4319,6 +4320,8 @@ static void virt_instance_init(Object *obj) vms->oem_id =g_strndup (ACPI_BUILD_APPNAME6 ,6 ); vms->oem_table_id =g_strndup (ACPI_BUILD_APPNAME8 ,8 );cxl_machine_init (obj, &vms->cxl_devices_state); + + vms->smmuv3_devices =g_ptr_array_new_with_free_func (NULL ); }static void virt_instance_finalize (Object *obj)
Signature · 1 line
2.43.0
PATCH v6 10/31 hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support
In reply to Shameer KolothumUse IOMMU_GET_HW_INFO to query host support for Tegra241 CMDQV.
Validate the returned data type, version, and minimum number of vCMDQs and SIDs per Tegra241 CMDQ Virtual Interface(VI). Fail the probe if the host does not meet these requirements.
The QEMU model supports one Virtual Interface(VI) per VM with 2 vCMDQs and 16 SIDs per VI, so the probe ensures the host implementation is compatible with these limits.
--- hw/arm/tegra241-cmdqv.h | 4 ++++ hw/arm/tegra241-cmdqv.c | 32 ++++++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.h +4 -0
@@ -10,6 +10,10 @@#ifndef HW_ARM_TEGRA241_CMDQV_H #define HW_ARM_TEGRA241_CMDQV_H +#define CMDQV_VER 1 +#define CMDQV_NUM_CMDQ_LOG2 1 +#define CMDQV_NUM_SID_PER_VI_LOG2 4 +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +30 -2
@@ -38,8 +38,36 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)static bool tegra241_cmdqv_probe (SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp) { -error_setg (errp,"NVIDIA Tegra241 CMDQV is unsupported" ); -return false; +uint32_t data_type =IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV ; +struct iommu_hw_info_tegra241_cmdqv cmdqv_info; +uint64_t caps; + +if (!iommufd_backend_get_device_info (idev->iommufd, idev->devid, &data_type, + &cmdqv_info,sizeof (cmdqv_info), &caps, +NULL , errp)) { +return false; + } +if (data_type !=IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV ) { +error_setg (errp,"Host CMDQV: unexpected data type %u (expected %u)" , + data_type,IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV ); +return false; + } +if (cmdqv_info.version !=CMDQV_VER ) { +error_setg (errp,"Host CMDQV: unsupported version %u (expected %u)" , + cmdqv_info.version,CMDQV_VER ); +return false; + } +if (cmdqv_info.log2vcmdqs <CMDQV_NUM_CMDQ_LOG2 ) { +error_setg (errp,"Host CMDQV: insufficient vCMDQs log2=%u (need >= %u)" , + cmdqv_info.log2vcmdqs,CMDQV_NUM_CMDQ_LOG2 ); +return false; + } +if (cmdqv_info.log2vsids <CMDQV_NUM_SID_PER_VI_LOG2 ) { +error_setg (errp,"Host CMDQV: insufficient SIDs log2=%u (need >= %u)" , + cmdqv_info.log2vsids,CMDQV_NUM_SID_PER_VI_LOG2 ); +return false; + } +return true; }static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = {
Signature · 1 line
2.43.0
PATCH v6 11/31 hw/arm/tegra241-cmdqv: Implement CMDQV init
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Tegra241 CMDQV extends SMMUv3 with support for virtual command queues (VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:
0x00000 (CMDQ-V Config page) 0x10000 (CMDQ-V CMDQ Page0) 0x20000 (CMDQ-V CMDQ Page1) 0x30000 (Virtual Interface Page0) 0x40000 (Virtual Interface Page1)
This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.
The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.
The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.
--- hw/arm/smmuv3-accel.h | 1 + hw/arm/tegra241-cmdqv.h | 20 ++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 40 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 59 insertions(+), 2 deletions(-)
hw/arm/smmuv3-accel.h +1 -0
@@ -61,6 +61,7 @@ typedef struct SMMUv3AccelState {bool auto_mode;bool auto_finalised;const SMMUv3AccelCmdqvOps *cmdqv_ops; +void *cmdqv;/* vendor specific CMDQV state */ } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
hw/arm/tegra241-cmdqv.h +20 -0
@@ -14,6 +14,26 @@#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4 +/* +* Tegra241 CMDQV MMIO layout (64KB pages) +* +* 0x00000 (CMDQ-V Config page) +* 0x10000 (CMDQ-V CMDQ Page0) +* 0x20000 (CMDQ-V CMDQ Page1) +* 0x30000 (Virtual Interface Page0) +* 0x40000 (Virtual Interface Page1) +*/ +#define TEGRA241_CMDQV_IO_LEN 0x50000 + +struct iommu_viommu_tegra241_cmdqv ; + +typedef struct Tegra241CMDQV { +struct iommu_viommu_tegra241_cmdqv *cmdqv_data; +SMMUv3AccelState *s_accel; +MemoryRegion mmio_cmdqv; +qemu_irq irq; +}Tegra241CMDQV ; +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +38 -2
@@ -13,6 +13,17 @@#include "smmuv3-accel.h" #include "tegra241-cmdqv.h" +static uint64_t tegra241_cmdqv_read_mmio (void *opaque,hwaddr offset, +unsigned size) +{ +return 0 ; +} + +static void tegra241_cmdqv_write_mmio (void *opaque,hwaddr offset, +uint64_t value,unsigned size) +{ +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) { }
@@ -29,10 +40,35 @@ static void tegra241_cmdqv_reset(SMMUv3State *s) { } +static const MemoryRegionOps mmio_cmdqv_ops = { + .read = tegra241_cmdqv_read_mmio, + .write = tegra241_cmdqv_write_mmio, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; +static bool tegra241_cmdqv_init (SMMUv3State *s,Error **errp) { -error_setg (errp,"NVIDIA Tegra241 CMDQV is unsupported" ); -return false; +SysBusDevice *sbd =SYS_BUS_DEVICE (OBJECT (s)); +SMMUv3AccelState *accel = s->s_accel; +Tegra241CMDQV *cmdqv; + + cmdqv =g_new0 (Tegra241CMDQV,1 ); + cmdqv->cmdqv_data = g_new0(struct iommu_viommu_tegra241_cmdqv, 1); +memory_region_init_io (&cmdqv->mmio_cmdqv,OBJECT (s), &mmio_cmdqv_ops, cmdqv, +"tegra241-cmdqv" ,TEGRA241_CMDQV_IO_LEN ); +sysbus_init_mmio (sbd, &cmdqv->mmio_cmdqv); +sysbus_init_irq (sbd, &cmdqv->irq); + cmdqv->s_accel = accel; + accel->cmdqv = cmdqv; +return true; } static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
Signature · 1 line
2.43.0
PATCH v6 12/31 hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus
In reply to Shameer KolothumSMMUv3 devices with acceleration may enable CMDQV extensions after device realize. In that case, additional MMIO regions and IRQ lines may be registered but not yet mapped to the platform bus.
Ensure SMMUv3 device resources are linked to the platform bus during machine_done().
This is safe to do unconditionally since the platform bus helpers skip resources that are already mapped.
--- hw/arm/virt.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
hw/arm/virt.c +22 -0
@@ -2361,6 +2361,25 @@ static void virt_build_smbios(VirtMachineState *vms) } } +/* +* SMMUv3 devices with acceleration may enable CMDQV extensions +* after device realize. In that case, additional MMIO regions and +* IRQ lines may be registered but not yet mapped to the platform bus. +* +* Ensure all resources are linked to the platform bus before final +* machine setup. +*/ + +static void virt_smmuv3_dev_link_cmdqv (VirtMachineState *vms) +{ +for (int i =0 ; i < vms->smmuv3_devices->len; i++) { +DeviceState *dev =g_ptr_array_index (vms->smmuv3_devices, i); + +platform_bus_link_device (PLATFORM_BUS_DEVICE (vms->platform_bus_dev), +SYS_BUS_DEVICE (dev)); + } +} +static void virt_machine_done (Notifier *notifier,void *data) {
@@ -2377,6 +2396,9 @@ void virt_machine_done(Notifier *notifier, void *data)if (vms->cxl_devices_state.is_enabled) {cxl_fmws_link_targets (&error_fatal); } + +virt_smmuv3_dev_link_cmdqv (vms); +/* * If the user provided a dtb, we assume the dynamic sysbus nodes * already are integrated there. This corresponds to a use case where
Signature · 1 line
2.43.0
PATCH v6 13/31 hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Replace the stub implementation with real vIOMMU allocation for Tegra241 CMDQV.
Allocate a matching vEVENTQ together with the vIOMMU, since it is specific to the Tegra241 CMDQV vIOMMU and used to receive CMDQV events.
Free both objects on teardown.
--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 48 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.h +1 -0
@@ -32,6 +32,7 @@ typedef struct Tegra241CMDQV {SMMUv3AccelState *s_accel;MemoryRegion mmio_cmdqv;qemu_irq irq; +IOMMUFDVeventq *veventq; } Tegra241CMDQV;const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );
hw/arm/tegra241-cmdqv.c +47 -1
66 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v6 14/31 hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Tegra241 CMDQV exposes control and status registers in the CMDQ-V Config page (offset [0x0, 0x10000)) used to configure virtual command queue allocation and interrupt behavior.
Add read/write emulation for the CMDQ-V Config region ([CMDQV_BASE, CMDQV_CMDQ_BASE]), backed by a simple register cache. This includes CONFIG, PARAM, STATUS, VI error and interrupt maps, CMDQ allocation map and the VINTF0 related registers defined in the CMDQ-V Config space. Only VINTF0 is supported; VINTF1-63 are not.
Dispatch writes on access size: Introduced writel_mmio for 4-byte and writell_mmio for 8-byte. Reads need no split as the MMIO framework masks the returned value to the access size.
--- hw/arm/tegra241-cmdqv.h | 110 ++++++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 181 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 4 + 3 files changed, 294 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.h +110 -0
@@ -10,10 +10,15 @@#ifndef HW_ARM_TEGRA241_CMDQV_H #define HW_ARM_TEGRA241_CMDQV_H +#include "hw/core/registerfields.h" +#define CMDQV_VER 1#define CMDQV_NUM_CMDQ_LOG2 1#define CMDQV_NUM_SID_PER_VI_LOG2 4 +#define TEGRA241_CMDQV_MAX_CMDQ (1U << CMDQV_NUM_CMDQ_LOG2) +#define TEGRA241_CMDQV_MAX_NUM_SID (1U << CMDQV_NUM_SID_PER_VI_LOG2) +/* * Tegra241 CMDQV MMIO layout (64KB pages) *
@@ -33,8 +38,113 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; + +/* CMDQ-V Config page register cache */ +uint32_t config; +uint32_t param; +uint32_t status; +uint32_t vi_err_map[2 ]; +uint32_t vi_int_mask[2 ]; +uint32_t cmdq_err_map[4 ]; +uint32_t cmdq_alloc_map[TEGRA241_CMDQV_MAX_CMDQ ]; + +/* VINTF0 register cache (within CMDQ-V Config page) */ +uint32_t vintf_config; +uint32_t vintf_status; +uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID ]; +uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID ]; +uint32_t vintf_cmdq_err_map[4 ]; } Tegra241CMDQV; +/* CMDQ-V Config page registers (offset 0x00000) */ +REG32(CONFIG, 0x0) +FIELD(CONFIG, CMDQV_EN, 0, 1) +FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3) +FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8) +FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8) +FIELD(CONFIG, CONS_DRAM_EN, 20, 1) + +REG32(PARAM, 0x4) +FIELD(PARAM, CMDQV_VER, 0, 4) +FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4) +FIELD(PARAM, CMDQV_NUM_VI_LOG2, 8, 4) +FIELD(PARAM, CMDQV_NUM_SID_PER_VI_LOG2, 12, 4) + +REG32(STATUS, 0x8) +FIELD (STATUS ,CMDQV_ENABLED ,0 ,1 ) + +/* SMMU_CMDQV_VI_ERR_MAP_0/1 definitions */ +#define A_VI_ERR_MAP_0 0x14 +#define A_VI_ERR_MAP_1 0x18 +#define V_VI_ERR_MAP_NO_ERROR (0) +#define V_VI_ERR_MAP_ERROR (1) + +/* SMMU_CMDQV_VI_INT_MASK_0/1 definitions */ +#define A_VI_INT_MASK_0 0x1c +#define A_VI_INT_MASK_1 0x20 +#define V_VI_INT_MASK_NOT_MASKED (0) +#define V_VI_INT_MASK_MASKED (1) + +/* SMMU_CMDQV_CMDQ_ERR_MAP_0-3 definitions */ +#define A_CMDQ_ERR_MAP_0 0x24 +#define A_CMDQ_ERR_MAP_1 0x28 +#define A_CMDQ_ERR_MAP_2 0x2c +#define A_CMDQ_ERR_MAP_3 0x30 + +/* +* CMDQ_ALLOC_MAP: one entry per physical VCMDQ. Hardware supports up to 128 +* entries (CMDQV_NUM_CMDQ_LOG2=7), but QEMU only exposes +* TEGRA241_CMDQV_MAX_CMDQ (=2) VCMDQs per VM so only entries 0 and 1 are +* defined here. +*/ +/* 2 identical register entries */ +#define SMMU_CMDQV_CMDQ_ALLOC_MAP_ (i) \ + REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4) \ + FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1) \ + FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7) \ + FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6) + +SMMU_CMDQV_CMDQ_ALLOC_MAP_(0) +SMMU_CMDQV_CMDQ_ALLOC_MAP_ (1 ) + +/* SMMU_CMDQV_VINTF0 registers (only VINTF0 is exposed to the guest) */ +REG32(VINTF0_CONFIG, 0x1000) +FIELD(VINTF0_CONFIG, ENABLE, 0, 1) +FIELD(VINTF0_CONFIG, VMID, 1, 16) +FIELD(VINTF0_CONFIG, HYP_OWN, 17, 1) + +REG32(VINTF0_STATUS, 0x1004) +FIELD(VINTF0_STATUS, ENABLE_OK, 0, 1) +FIELD(VINTF0_STATUS, STATUS, 1, 3) +FIELD(VINTF0_STATUS, VI_NUM_LVCMDQ, 16, 8) + +#define V_VINTF_STATUS_NO_ERROR (0 << 1) +#define V_VINTF_STATUS_VCMDQ_ERROR (1 << 1) + +/* +* SMMU_CMDQV_VINTF0_SID_MATCH/_REPLACE: 16 entries per VINTF +* (CMDQV_NUM_SID_PER_VI_LOG2=4). Only _0 and _15 are defined, +* used as switch case range bounds. +*/ +REG32(VINTF0_SID_MATCH_0, 0x1040) +FIELD(VINTF0_SID_MATCH_0, ENABLE, 0, 1) +FIELD(VINTF0_SID_MATCH_0, VIRT_SID, 1, 20) +#define A_VINTF0_SID_MATCH_15 (A_VINTF0_SID_MATCH_0 + 15 * 4) + +REG32(VINTF0_SID_REPLACE_0, 0x1080) +FIELD(VINTF0_SID_REPLACE_0, PHYS_SID, 0, 20) +#define A_VINTF0_SID_REPLACE_15 (A_VINTF0_SID_REPLACE_0 + 15 * 4) + +/* +* SMMU_CMDQV_VINTF0_LVCMDQ_ERR_MAP: 4 registers per VINTF covering 32 logical +* VCMDQs each. With TEGRA241_CMDQV_MAX_CMDQ=2, only MAP_0 bits [1:0] carry +* error state. MAP_1..MAP_3 always read as 0. Only _0 and _3 are defined, +* used as switch case range bounds. +*/ +REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) +FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32) +#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4) +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +180 -1
201 lines not on this page · raw message
hw/arm/trace-events +4 -0
@@ -72,6 +72,10 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x" +# tegra241-cmdqv +tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" + # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d" strongarm_ssp_read_underrun(void) "SSP rx underrun"
Signature · 1 line
2.43.0
PATCH v6 15/31 hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Tegra241 CMDQV exposes per-VCMDQ register windows through two MMIO apertures:
Direct VCMDQ aperture (0x10000/0x20000): VCMDQ Page0/Page1 VINTF logical aperture (0x30000/0x40000): VINTF0 LVCMDQ Page0/Page1
Both apertures are hardware aliases of the same underlying registers:
Page 0 (control/status): CONS_INDX, PROD_INDX, CONFIG, STATUS,
GERROR, GERRORN
Page 1 (base/DRAM): BASE_L/H, CONS_INDX_BASE_DRAM_L/H
The direct aperture Page 0 is programmable at any time so long as CMDQV_EN is enabled. The VINTF (logical) aperture Page 0 is programmable only once SW has mapped a VCMDQ to a VINTF; the "logical" view is local to that VINTF.
Add read emulation for both apertures, backed by a single per-VCMDQ register cache. VINTF aperture reads are translated to their equivalent direct-aperture offset and served from the same cached state.
Per the CMDQV architecture, a VCMDQ must be allocated to a Virtual Interface before it is used to send commands to the SMMU. Until that allocation happens, reads return cached register state with no HW interaction. Subsequent patches wire up IOMMU_HW_QUEUE_ALLOC, mmap the host VINTF Page 0, and install it into guest MMIO; after that, Page 0 reads from either aperture are served from the hardware-backed mmap'd page instead of the cache. Page 1 is also a hardware alias, but the kernel only exposes mmap for Page 0, so Page 1 reads always trap to QEMU and are served from cache.
--- hw/arm/tegra241-cmdqv.h | 216 ++++++++++++++++++++++++++++++++++++++++ hw/arm/tegra241-cmdqv.c | 103 +++++++++++++++++++ hw/arm/trace-events | 2 + 3 files changed, 321 insertions(+)
hw/arm/tegra241-cmdqv.h +216 -0
@@ -30,6 +30,13 @@*/ #define TEGRA241_CMDQV_IO_LEN 0x50000 +/* CMDQV MMIO aperture bases and VCMDQ stride */ +#define CMDQV_VCMDQ_PAGE0_BASE 0x10000/* CMDQV_CMDQ_BASE */ +#define CMDQV_VCMDQ_PAGE1_BASE 0x20000 +#define CMDQV_VINTF_PAGE0_BASE 0x30000/* CMDQV_VI_CMDQ_BASE */ +#define CMDQV_VINTF_PAGE1_BASE 0x40000 +#define CMDQV_VCMDQ_STRIDE 0x80 +struct iommu_viommu_tegra241_cmdqv ; typedef struct Tegra241CMDQV {
@@ -54,6 +61,19 @@ typedef struct Tegra241CMDQV {uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID ];uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID ];uint32_t vintf_cmdq_err_map[4 ]; +/* +* VCMDQ register cache. The direct (VCMDQ aperture) and logical +* (VINTF aperture) views are hardware aliases; both are served from +* this single cached copy. +*/ +uint32_t vcmdq_cons_indx[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_prod_indx[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_config[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_status[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_gerror[TEGRA241_CMDQV_MAX_CMDQ ]; +uint32_t vcmdq_gerrorn[TEGRA241_CMDQV_MAX_CMDQ ]; +uint64_t vcmdq_base[TEGRA241_CMDQV_MAX_CMDQ ]; +uint64_t vcmdq_cons_indx_base[TEGRA241_CMDQV_MAX_CMDQ ]; } Tegra241CMDQV;/* CMDQ-V Config page registers (offset 0x00000) */
@@ -145,6 +165,202 @@ REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4) +/* +* Direct VCMDQ aperture register windows. +* +* Page 0 @ CMDQV_VCMDQ_PAGE0_BASE: VCMDQ control and status registers. +* Page 1 @ CMDQV_VCMDQ_PAGE1_BASE: VCMDQ base and DRAM address registers. +* +* Each VCMDQ occupies a CMDQV_VCMDQ_STRIDE-byte slot within its page. +*/ + +/* --- Page 0 register macros --- */ +#define SMMU_CMDQV_VCMDQi_CONS_INDX_ (i) \ + REG32(VCMDQ##i##_CONS_INDX, \ + CMDQV_VCMDQ_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20) \ + FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7) + +#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE 0 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE 1 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT 2 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3 +#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS 4 + +#define SMMU_CMDQV_VCMDQi_PROD_INDX_ (i) \ + REG32(VCMDQ##i##_PROD_INDX, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20) + +#define SMMU_CMDQV_VCMDQi_CONFIG_ (i) \ + REG32(VCMDQ##i##_CONFIG, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1) + +#define SMMU_CMDQV_VCMDQi_STATUS_ (i) \ + REG32(VCMDQ##i##_STATUS, \ + CMDQV_VCMDQ_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1) + +#define SMMU_CMDQV_VCMDQi_GERROR_ (i) \ + REG32(VCMDQ##i##_GERROR, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1) + +#define SMMU_CMDQV_VCMDQi_GERRORN_ (i) \ + REG32(VCMDQ##i##_GERRORN, \ + CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +/* Page 0 layout: VCMDQ0 */ +SMMU_CMDQV_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VCMDQi_STATUS_(0) +SMMU_CMDQV_VCMDQi_GERROR_(0) +SMMU_CMDQV_VCMDQi_GERRORN_ (0 ) + +/* Page 0 layout: VCMDQ1 */ +SMMU_CMDQV_VCMDQi_CONS_INDX_(1) +SMMU_CMDQV_VCMDQi_PROD_INDX_(1) +SMMU_CMDQV_VCMDQi_CONFIG_(1) +SMMU_CMDQV_VCMDQi_STATUS_(1) +SMMU_CMDQV_VCMDQi_GERROR_(1) +SMMU_CMDQV_VCMDQi_GERRORN_ (1 ) + +/* --- Page 1 register macros --- */ +#define SMMU_CMDQV_VCMDQi_BASE_L_ (i) \ + REG32(VCMDQ##i##_BASE_L, CMDQV_VCMDQ_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5) \ + FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27) + +#define SMMU_CMDQV_VCMDQi_BASE_H_ (i) \ + REG32(VCMDQ##i##_BASE_H, \ + CMDQV_VCMDQ_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16) + +#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_ (i) \ + REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, \ + CMDQV_VCMDQ_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32) + +#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (i) \ + REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, \ + CMDQV_VCMDQ_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16) + +/* Page 1 layout: VCMDQ0 */ +SMMU_CMDQV_VCMDQi_BASE_L_(0) +SMMU_CMDQV_VCMDQi_BASE_H_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(0) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (0 ) + +/* Page 1 layout: VCMDQ1 */ +SMMU_CMDQV_VCMDQi_BASE_L_(1) +SMMU_CMDQV_VCMDQi_BASE_H_(1) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) +SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_ (1 ) + +/* +* VINTF0 logical VCMDQ aperture register windows. +* +* Page 0 @ CMDQV_VINTF_PAGE0_BASE: VCMDQ control and status registers. +* Page 1 @ CMDQV_VINTF_PAGE1_BASE: VCMDQ base and DRAM address registers. +* +* VCMDQs mapped via VINTF are accessed through this aperture as +* hardware aliases of the direct VCMDQ aperture above. +*/ + +/* --- Page 0 register macros --- */ +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX, \ + CMDQV_VINTF_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20) \ + FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7) + +#define SMMU_CMDQV_VI_VCMDQi_PROD_INDX_ (i) \ + REG32(VI_VCMDQ##i##_PROD_INDX, \ + CMDQV_VINTF_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20) + +#define SMMU_CMDQV_VI_VCMDQi_CONFIG_ (i) \ + REG32(VI_VCMDQ##i##_CONFIG, \ + CMDQV_VINTF_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1) + +#define SMMU_CMDQV_VI_VCMDQi_STATUS_ (i) \ + REG32(VI_VCMDQ##i##_STATUS, \ + CMDQV_VINTF_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1) + +#define SMMU_CMDQV_VI_VCMDQi_GERROR_ (i) \ + REG32(VI_VCMDQ##i##_GERROR, \ + CMDQV_VINTF_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1) \ + FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1) + +#define SMMU_CMDQV_VI_VCMDQi_GERRORN_ (i) \ + REG32(VI_VCMDQ##i##_GERRORN, \ + CMDQV_VINTF_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1) \ + FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \ + FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1) + +/* Page 0 layout: VCMDQ0 */ +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(0) +SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(0) +SMMU_CMDQV_VI_VCMDQi_CONFIG_(0) +SMMU_CMDQV_VI_VCMDQi_STATUS_(0) +SMMU_CMDQV_VI_VCMDQi_GERROR_(0) +SMMU_CMDQV_VI_VCMDQi_GERRORN_ (0 ) + +/* Page 0 layout: VCMDQ1 */ +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(1) +SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(1) +SMMU_CMDQV_VI_VCMDQi_CONFIG_(1) +SMMU_CMDQV_VI_VCMDQi_STATUS_(1) +SMMU_CMDQV_VI_VCMDQi_GERROR_(1) +SMMU_CMDQV_VI_VCMDQi_GERRORN_ (1 ) + +/* --- Page 1 register macros --- */ +#define SMMU_CMDQV_VI_VCMDQi_BASE_L_ (i) \ + REG32(VI_VCMDQ##i##_BASE_L, \ + CMDQV_VINTF_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5) \ + FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27) + +#define SMMU_CMDQV_VI_VCMDQi_BASE_H_ (i) \ + REG32(VI_VCMDQ##i##_BASE_H, \ + CMDQV_VINTF_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16) + +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, \ + CMDQV_VINTF_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32) + +#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_ (i) \ + REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, \ + CMDQV_VINTF_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \ + FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16) + +/* Page 1 layout: VCMDQ0 */ +SMMU_CMDQV_VI_VCMDQi_BASE_L_(0) +SMMU_CMDQV_VI_VCMDQi_BASE_H_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(0) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_ (0 ) + +/* Page 1 layout: VCMDQ1 */ +SMMU_CMDQV_VI_VCMDQi_BASE_L_(1) +SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) +SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +103 -0
121 lines not on this page · raw message
hw/arm/trace-events +2 -0
@@ -75,6 +75,8 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
PATCH v6 16/31 hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache.
Per the CMDQV architecture, a VCMDQ must be allocated to a Virtual Interface before it is used to send commands to the SMMU. Until that allocation happens, MMIO writes only update cached register state - no command consumption, error handling, or interrupt activity is driven from these writes. Subsequent patches wire up IOMMU_HW_QUEUE_ALLOC, mmap the host VINTF Page 0, and install it into guest MMIO; after that, Page 0 writes from either aperture reach the hardware-backed mmap'd page instead of just the cache.
--- hw/arm/tegra241-cmdqv.c | 159 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 2 + 2 files changed, 159 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +157 -2
@@ -120,6 +120,104 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, } } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/ +static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value,bool direct) +{ +switch (offset0) { +case A_VCMDQ0_CONS_INDX : + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : +/* VCMDQ is functional only once allocated to a VINTF; cache only. */ + cmdqv->vcmdq_prod_indx[index] = value; +break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }else { + cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : +/* VCMDQ is functional only once allocated to a VINTF; cache only. */ + cmdqv->vcmdq_gerrorn[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, direct ?"direct" :"vi" , + offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value,bool direct) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_H : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],32 ,32 , value); +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, direct ?"direct" :"vi" , + offset0, value); +} + +/* +* Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L. +*/ +static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint64_t value,bool direct) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : + cmdqv->vcmdq_base[index] = value; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled 64-bit write at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, direct ?"direct" :"vi" , + offset0, value); +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) {
@@ -243,6 +341,8 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +int index; +switch (offset) {case A_CONFIG : cmdqv->config = value;
@@ -261,6 +361,39 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value);break ; + case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN: +/* +* VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers. +* Translate the VINTF aperture offset to its VCMDQ Page0 equivalent +* before dispatching to the Page 0 helper. +*/ + offset -=CMDQV_VINTF_PAGE0_BASE -CMDQV_VCMDQ_PAGE0_BASE ; + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); +break ; + case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN: +/* +* Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a +* CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize +* to the VCMDQ0_* offset before calling the Page 0 helper. +*/ + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); +break ; + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); +break ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); +break ;default :qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset);
@@ -268,14 +401,36 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, }/* -* 8-byte MMIO write handler. +* 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept +* full 64-bit writes; other offsets are write-ignored. */ static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { -qemu_log_mask (LOG_UNIMP , +int index; + +switch (offset) { + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* +* VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers. +* Translate the VINTF aperture offset to its VCMDQ Page1 equivalent +* before dispatching to the Page 1 helper. +*/ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); +break ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +2 -0
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
Re: [PATCH v6 16/31] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes
Quoted 20 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.
VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE,
CONS_INDX_BASE) always update the cache.Per the CMDQV architecture, a VCMDQ must be allocated to a Virtual Interface before it is used to send commands to the SMMU. Until that allocation happens, MMIO writes only update cached register state - no command consumption, error handling, or interrupt activity is driven from these writes. Subsequent patches wire up
IOMMU_HW_QUEUE_ALLOC, mmap the host VINTF Page 0, and install it into guest MMIO; after that, Page 0 writes from either aperture reach the hardware-backed mmap'd page instead of just the cache.
Quoted 3 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Thanks
Eric
Quoted 215 lines
--- hw/arm/tegra241-cmdqv.c | 159 +++++++++++++++++++++++++++++++++++++++- hw/arm/trace-events | 2 + 2 files changed, 159 insertions(+), 2 deletions(-)hw/arm/tegra241-cmdqv.c +157
-2@@ -120,6 +120,104 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, } } +/* +* Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. +* +* The caller normalizes the MMIO offset such that @offset0 always refers +* to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* Page 0 registers are all 32-bit; this helper is only called for 4-byte +* writes. +*/ +static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value,bool direct) +{ +switch (offset0) { +case A_VCMDQ0_CONS_INDX : + cmdqv->vcmdq_cons_indx[index] = value; +break ; +case A_VCMDQ0_PROD_INDX : +/* VCMDQ is functional only once allocated to a VINTF; cache only. */ + cmdqv->vcmdq_prod_indx[index] = value; +break ; +case A_VCMDQ0_CONFIG : +if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { + cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }else { + cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + } + cmdqv->vcmdq_config[index] = value; +break ; +case A_VCMDQ0_GERRORN : +/* VCMDQ is functional only once allocated to a VINTF; cache only. */ + cmdqv->vcmdq_gerrorn[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page0 (index, direct ?"direct" :"vi" , + offset0, value); +} + +/* +* Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access. +*/ +static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint32_t value,bool direct) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_BASE_H : + cmdqv->vcmdq_base[index] = +deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],0 ,32 , value); +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_H : + cmdqv->vcmdq_cons_indx_base[index] = +deposit64 (cmdqv->vcmdq_cons_indx_base[index],32 ,32 , value); +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled write access at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, direct ?"direct" :"vi" , + offset0, value); +} + +/* +* Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L. +*/ +static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv, +hwaddr offset0,int index, +uint64_t value,bool direct) +{ +switch (offset0) { +case A_VCMDQ0_BASE_L : + cmdqv->vcmdq_base[index] = value; +break ; +case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : + cmdqv->vcmdq_cons_indx_base[index] = value; +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s unhandled 64-bit write at 0x%" PRIx64"\n" , + __func__, offset0); + } +trace_tegra241_cmdqv_write_vcmdq_page1 (index, direct ?"direct" :"vi" , + offset0, value); +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) {@@ -243,6 +341,8 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +int index; +switch (offset) {case A_CONFIG : cmdqv->config = value;@@ -261,6 +361,39 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value);break ; + case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN: +/* +* VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers. +* Translate the VINTF aperture offset to its VCMDQ Page0 equivalent +* before dispatching to the Page 0 helper. +*/ + offset -=CMDQV_VINTF_PAGE0_BASE -CMDQV_VCMDQ_PAGE0_BASE ; + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); +break ; + case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN: +/* +* Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a +* CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize +* to the VCMDQ0_* offset before calling the Page 0 helper. +*/ + index = (offset -CMDQV_VCMDQ_PAGE0_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page0 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); +break ; + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); +break ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); +break ;default :qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset);@@ -268,14 +401,36 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, }/* -* 8-byte MMIO write handler. +* 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept +* full 64-bit writes; other offsets are write-ignored. */ static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { -qemu_log_mask (LOG_UNIMP , +int index; + +switch (offset) { + case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H: +/* +* VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers. +* Translate the VINTF aperture offset to its VCMDQ Page1 equivalent +* before dispatching to the Page 1 helper. +*/ + offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); +break ; + case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: + index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ; +tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); +break ; +default : +qemu_log_mask (LOG_UNIMP ,"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,hw/arm/trace-events +2
-0@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 +tegra241_cmdqv_write_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 # strongarm.c strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
PATCH v6 17/31 hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
The kernel currently exposes a single VINTF per emulated SMMUv3 instance. IOMMU_VIOMMU_ALLOC returns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF after IOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.
When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.
--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
@@ -37,6 +37,8 @@#define CMDQV_VINTF_PAGE1_BASE 0x40000#define CMDQV_VCMDQ_STRIDE 0x80 +#define VINTF_PAGE_SIZE 0x10000 +struct iommu_viommu_tegra241_cmdqv ; typedef struct Tegra241CMDQV {
@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +void *vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;
hw/arm/tegra241-cmdqv.c +47 -3
88 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer,
Quoted 100 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The kernel currently exposes a single VINTF per emulated SMMUv3 instance.
IOMMU_VIOMMU_ALLOCreturns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF afterIOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 50 insertions(+), 3 deletions(-)hw/arm/tegra241-cmdqv.h +3
-0@@ -37,6 +37,8 @@#define CMDQV_VINTF_PAGE1_BASE 0x40000#define CMDQV_VCMDQ_STRIDE 0x80 +#define VINTF_PAGE_SIZE 0x10000 +struct iommu_viommu_tegra241_cmdqv ; typedef struct Tegra241CMDQV {@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +void *vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c +37
-1@@ -9,6 +9,8 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "qemu/error-report.h" +#include "qapi/error.h" #include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" @@ -218,8 +220,42 @@ static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv, offset0, value); } +static bool +tegra241_cmdqv_munmap_vintf_page0 (Tegra241CMDQV *cmdqv,Error **errp) +{ +if (!cmdqv->vintf_page0) { +return true; + } + +if (munmap (cmdqv->vintf_page0,VINTF_PAGE_SIZE ) <0 ) { +error_setg_errno (errp, errno,"Failed to unmap VINTF page0" ); +return false; + } + cmdqv->vintf_page0 =NULL ; +return true; +} + +static bool tegra241_cmdqv_mmap_vintf_page0 (Tegra241CMDQV *cmdqv,Error **errp) +{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; + +if (cmdqv->vintf_page0) { +return true; + } + +if (!iommufd_backend_viommu_mmap (viommu->iommufd, viommu->viommu_id, +VINTF_PAGE_SIZE , + cmdqv->cmdqv_data->out_vintf_mmap_offset, + &cmdqv->vintf_page0, errp)) { +return false; + } + +return true; +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv, -hwaddr offset,uint64_t value) +hwaddr offset,uint64_t value, +Error **errp) {int i;
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 277
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +4
-1@@ -234,8 +270,11 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, cmdqv->vintf_config = value;if (value &R_VINTF0_CONFIG_ENABLE_MASK ) { - cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { + cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; + } }else { +tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp);
the munmap is only done upon guest action in this patch. Later on, it is also done on reset (in smmuv3_accel_reset called on smmu_reset_exit called in exit phase). So I think we are good overall but it is a bit annoying to leave it done only upon guest action in this patch. What do you think?
Thanks
Eric
Quoted 31 lines
hw/arm/tegra241-cmdqv.c +6
-1cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ; }break ;@@ -341,6 +380,7 @@ out:static void tegra241_cmdqv_writel_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint32_t value) { +Error *local_err =NULL ;int index;switch (offset) {@@ -359,7 +399,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, cmdqv->cmdq_alloc_map[(offset -A_CMDQ_ALLOC_MAP_0 ) /4 ] = value;break ; case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3: -tegra241_cmdqv_config_vintf_write (cmdqv, offset, value); +tegra241_cmdqv_config_vintf_write (cmdqv, offset, value, &local_err);break ; case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:/* @@ -398,6 +438,10 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,qemu_log_mask (LOG_UNIMP ,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset); } + +if (local_err) { +error_report_err (local_err); + } }/*
RE: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Eric,
Quoted 134 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:01
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 13 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The kernel currently exposes a single VINTF per emulated SMMUv3 instance.
IOMMU_VIOMMU_ALLOCreturns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF afterIOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.
Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 50++++++++++++++++++++++++++++++++++++++---
hw/arm/tegra241-cmdqv.c static void
Quoted 37 lines
2 files changed, 50 insertions(+), 3 deletions(-)hw/arm/tegra241-cmdqv.h +3
-0@@ -37,6 +37,8 @@#define CMDQV_VINTF_PAGE1_BASE 0x40000#define CMDQV_VCMDQ_STRIDE 0x80 +#define VINTF_PAGE_SIZE 0x10000 +struct iommu_viommu_tegra241_cmdqv ; typedef struct Tegra241CMDQV {@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +void *vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c +2
-0@@ -9,6 +9,8 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "qemu/error-report.h" +#include "qapi/error.h" #include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" @@ -218,8 +220,42 @@ static voidtegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +2
-0offset0, value); } +static bool +tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error**errp)
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+{ +if (!cmdqv->vintf_page0) { +return true; + }hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ +if (munmap (cmdqv->vintf_page0,VINTF_PAGE_SIZE ) <0 ) { +error_setg_errno (errp, errno,"Failed to unmap VINTF page0" ); +return false; + } + cmdqv->vintf_page0 =NULL ; +return true; +} + +static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV*cmdqv, Error **errp)
hw/arm/tegra241-cmdqv.c static void
Quoted 26 lines
hw/arm/tegra241-cmdqv.c +19
-1+{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; + +if (cmdqv->vintf_page0) { +return true; + } + +if (!iommufd_backend_viommu_mmap (viommu->iommufd, viommu- viommu_id, +VINTF_PAGE_SIZE , + cmdqv->cmdqv_data->out_vintf_mmap_offset, + &cmdqv->vintf_page0, errp)) { +return false; + } + +return true; +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv, -hwaddr offset,uint64_t value) +hwaddr offset,uint64_t value, +Error **errp) {int i;@@ -234,8 +270,11 @@ static voidtegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 277
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +4
-1cmdqv->vintf_config = value;if (value &R_VINTF0_CONFIG_ENABLE_MASK ) { - cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { + cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; + } }else { +tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp);
Quoted 5 lines
the munmap is only done upon guest action in this patch. Later on, it is also done on reset (in
smmuv3_accel_resetcalled onsmmu_reset_exitcalled in exit phase). So I think we are good overall but it is a bit annoying to leave it done only upon guest action in this patch. What do you think?
Though I didn’t really get the concern here, there is a suggestion to move the mmap just after IOMMU_VIOMMU_ALLOC as it doesn't need to be tied to VINTF enable.
https://lore.kernel.org/qemu-devel/[email protected]/T/#m972247a4652b09e3d86def921d9a8551b7620341
If we do that it will keep mapped until VIOMMU is alive. Does that address the concern above?
Thanks,
Shameer
Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Quoted 140 lines · Shameer Kolothum Thodi
Hi Eric,
Quoted 133 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:01
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQVExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 13 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
The kernel currently exposes a single VINTF per emulated SMMUv3 instance.
IOMMU_VIOMMU_ALLOCreturns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF afterIOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.
Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 50++++++++++++++++++++++++++++++++++++++---
hw/arm/tegra241-cmdqv.c static void
Quoted 37 lines
2 files changed, 50 insertions(+), 3 deletions(-)hw/arm/tegra241-cmdqv.h +3
-0@@ -37,6 +37,8 @@#define CMDQV_VINTF_PAGE1_BASE 0x40000#define CMDQV_VCMDQ_STRIDE 0x80 +#define VINTF_PAGE_SIZE 0x10000 +struct iommu_viommu_tegra241_cmdqv ; typedef struct Tegra241CMDQV {@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +void *vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c +2
-0@@ -9,6 +9,8 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "qemu/error-report.h" +#include "qapi/error.h" #include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" @@ -218,8 +220,42 @@ static voidtegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +2
-0offset0, value); } +static bool +tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error**errp)
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+{ +if (!cmdqv->vintf_page0) { +return true; + }hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ +if (munmap (cmdqv->vintf_page0,VINTF_PAGE_SIZE ) <0 ) { +error_setg_errno (errp, errno,"Failed to unmap VINTF page0" ); +return false; + } + cmdqv->vintf_page0 =NULL ; +return true; +} + +static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV*cmdqv, Error **errp)
hw/arm/tegra241-cmdqv.c static void
Quoted 26 lines
hw/arm/tegra241-cmdqv.c +19
-1+{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; + +if (cmdqv->vintf_page0) { +return true; + } + +if (!iommufd_backend_viommu_mmap (viommu->iommufd, viommu- viommu_id, +VINTF_PAGE_SIZE , + cmdqv->cmdqv_data->out_vintf_mmap_offset, + &cmdqv->vintf_page0, errp)) { +return false; + } + +return true; +} +static void tegra241_cmdqv_config_vintf_write (Tegra241CMDQV *cmdqv, -hwaddr offset,uint64_t value) +hwaddr offset,uint64_t value, +Error **errp) {int i;@@ -234,8 +270,11 @@ static voidtegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 277
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +4
-1cmdqv->vintf_config = value;if (value &R_VINTF0_CONFIG_ENABLE_MASK ) { - cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { + cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; + } }else { +tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp);Quoted 5 lines
the munmap is only done upon guest action in this patch. Later on, it is also done on reset (in
smmuv3_accel_resetcalled onsmmu_reset_exitcalled in exit phase). So I think we are good overall but it is a bit annoying to leave it done only upon guest action in this patch. What do you think?
Quoted 8 lines
Though I didn’t really get the concern here, there is a suggestion to move the mmap just after
IOMMU_VIOMMU_ALLOCas it doesn't need to be tied to VINTF enable.https://lore.kernel.org/qemu-devel/[email protected]/T/#m972247a4652b09e3d86def921d9a8551b7620341
If we do that it will keep mapped until VIOMMU is alive. Does that address the concern above?
my concern is the munmap shall be eventually done on exit even if the guest does not issue the associated command, no? This is done later in the patch that introduces the reset. My concern is about patch sequencing. I think this is unrelated to Nicolin's suggestion.
Thanks
Eric
Quoted 7 lines
Thanks,
Shameer
PATCH v6 18/31 hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
iommufd_backend_alloc_hw_queue() needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
On allocation failure, set CMDQ_INIT_ERR and clear CMDQ_EN_OK in the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.
--- hw/arm/tegra241-cmdqv.h | 11 +++ hw/arm/tegra241-cmdqv.c | 154 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 156 insertions(+), 9 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */
@@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +145 -9
245 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Hi Shameer,
Quoted 78 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
VCMDQ_ENlives inVCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
iommufd_backend_alloc_hw_queue()needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
On allocation failure, set
CMDQ_INIT_ERRand clearCMDQ_EN_OKin the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 11 +++ hw/arm/tegra241-cmdqv.c | 154 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 156 insertions(+), 9 deletions(-)hw/arm/tegra241-cmdqv.h +11
-0@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */ @@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +9
-0@@ -18,6 +18,96 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) +{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; +IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index]; + +if (!vcmdq) { +return ; + } +iommufd_backend_free_id (viommu->iommufd, vcmdq->hw_queue_id);
hw/arm/tegra241-cmdqv.c line 39
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+g_free (vcmdq); + cmdqv->vcmdq[index] =NULL ; +} + +static void tegra241_cmdqv_free_all_vcmdq (Tegra241CMDQV *cmdqv) +{ +/* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */ +for (int i = (TEGRA241_CMDQV_MAX_CMDQ -1 ); i >=0 ; i--) { +tegra241_cmdqv_free_vcmdq (cmdqv, i); + }
basically the same comment as previous patch. I would add in the exit phase reset at this stage too.
Quoted 28 lines
hw/arm/tegra241-cmdqv.c +28
-0+} + +/* +* Allocate a host HW VCMDQ from the current cached BASE / size for @index. +* +* Setup is a no-op (returns true) when any of the following preconditions +* isn't met yet: +* - BASE not prgrammed yet. +* - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0) +* - BASE / size don't resolve to a RAM region +* - CMDQV global enable or VINTF enable is not yet asserted +*/ +static bool tegra241_cmdqv_setup_vcmdq (Tegra241CMDQV *cmdqv,int index, +Error **errp) +{ +SMMUv3AccelState *accel = cmdqv->s_accel; +uint64_t base_mask = (uint64_t )R_VCMDQ0_BASE_L_ADDR_MASK | + (uint64_t )R_VCMDQ0_BASE_H_ADDR_MASK <<32 ; +uint64_t addr = cmdqv->vcmdq_base[index] & base_mask; +uint64_t log2 = cmdqv->vcmdq_base[index] &R_VCMDQ0_BASE_L_LOG2SIZE_MASK ; +uint64_t size =1ULL << (log2 +4 ); +IOMMUFDViommu *viommu = accel->viommu; +IOMMUFDHWqueue *hw_queue; +uint32_t hw_queue_id; + +/* BASE not programmed yet. */ +if (!cmdqv->vcmdq_base[index]) { +return true;
hw/arm/tegra241-cmdqv.c line 77
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ } + +/* VCMDQ not yet mapped to a VINTF. */ +if (!(cmdqv->cmdq_alloc_map[index] &R_CMDQ_ALLOC_MAP_0_ALLOC_MASK )) { +return true; + } + +if (!tegra241_cmdqv_enabled (cmdqv) || !tegra241_vintf_enabled (cmdqv)) { +return true; + }
nit: I think I would introduce an helper vcmdq_hw_queue_ready_to_be_allocated(cmdqv, index) where you could put a proper doc comment.
Not that once flag the base is set it returns true. ie. you do not check that both L & H are set.
But anyway the queue will be reallocated
Quoted 21 lines
hw/arm/tegra241-cmdqv.c +21
-0+ +tegra241_cmdqv_free_vcmdq (cmdqv, index); + +if (!iommufd_backend_alloc_hw_queue (viommu->iommufd, viommu->viommu_id, +IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV , + index, addr, size, &hw_queue_id, + errp)) { +/* Record the failure in the cache. */ + cmdqv->vcmdq_gerror[index] |=R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK ; + cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +return false; + } + hw_queue =g_new (IOMMUFDHWqueue,1 ); + hw_queue->hw_queue_id = hw_queue_id; + hw_queue->viommu = viommu; + cmdqv->vcmdq[index] = hw_queue; + + cmdqv->vcmdq_gerror[index] &= ~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK ; + cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + +return true;
hw/arm/tegra241-cmdqv.c line 108
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+} + +static void tegra241_cmdqv_setup_all_vcmdq (Tegra241CMDQV *cmdqv, +Error **errp) +{ +for (int i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { +if (!tegra241_cmdqv_setup_vcmdq (cmdqv, i, errp)) { +return ; + } + }
nit: you you could after the tegra241_cmdqv_free_all_vcmdq helper
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2
-0+} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. *
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 243
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +6
-1@@ -145,7 +235,12 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,break ;case A_VCMDQ0_CONFIG :if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { - cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +/* Report init error if any. */ +if (!(cmdqv->vcmdq_gerror[index] & +R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK )) { + cmdqv->vcmdq_status[index] |= +R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }
does it really belong to this patch?
Quoted 136 lines
hw/arm/tegra241-cmdqv.c +45
-8}else { cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; }@@ -169,16 +264,19 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,*/ static void tegra241_cmdqv_write_vcmdq_page1 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index, -uint32_t value,bool direct) +uint32_t value,bool direct, +Error **errp) {switch (offset0) {case A_VCMDQ0_BASE_L : cmdqv->vcmdq_base[index] =deposit64 (cmdqv->vcmdq_base[index],0 ,32 , value); +tegra241_cmdqv_setup_vcmdq (cmdqv, index, errp);break ;case A_VCMDQ0_BASE_H : cmdqv->vcmdq_base[index] =deposit64 (cmdqv->vcmdq_base[index],32 ,32 , value); +tegra241_cmdqv_setup_vcmdq (cmdqv, index, errp);break ;case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : cmdqv->vcmdq_cons_indx_base[index] =@@ -202,11 +300,13 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,*/ static void tegra241_cmdqv_write_vcmdq_page1_64 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index, -uint64_t value,bool direct) +uint64_t value,bool direct, +Error **errp) {switch (offset0) {case A_VCMDQ0_BASE_L : cmdqv->vcmdq_base[index] = value; +tegra241_cmdqv_setup_vcmdq (cmdqv, index, errp);break ;case A_VCMDQ0_CONS_INDX_BASE_DRAM_L : cmdqv->vcmdq_cons_indx_base[index] = value;@@ -272,8 +372,14 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,if (value &R_VINTF0_CONFIG_ENABLE_MASK ) {if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +/* +* VCMDQs whose BASE was programmed before VINTF was +* enabled need their hw_queue allocated now. +*/ +tegra241_cmdqv_setup_all_vcmdq (cmdqv, errp); } }else { +tegra241_cmdqv_free_all_vcmdq (cmdqv);tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ; }@@ -388,16 +494,37 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, cmdqv->config = value;if (value &R_CONFIG_CMDQV_EN_MASK ) { cmdqv->status |=R_STATUS_CMDQV_ENABLED_MASK ; +/* +* VCMDQs whose BASE was programmed before CMDQV was enabled +* need their hw_queue allocated now. +*/ +tegra241_cmdqv_setup_all_vcmdq (cmdqv, &local_err); }else { +tegra241_cmdqv_free_all_vcmdq (cmdqv); cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK ; }break ; case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1: cmdqv->vi_int_mask[(offset -A_VI_INT_MASK_0 ) /4 ] = value;break ; - case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: - cmdqv->cmdq_alloc_map[(offset -A_CMDQ_ALLOC_MAP_0 ) /4 ] = value; + case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: { +int idx = (offset -A_CMDQ_ALLOC_MAP_0 ) /4 ; +bool was_alloc = cmdqv->cmdq_alloc_map[idx] & +R_CMDQ_ALLOC_MAP_0_ALLOC_MASK ; +bool now_alloc = value &R_CMDQ_ALLOC_MAP_0_ALLOC_MASK ; + + cmdqv->cmdq_alloc_map[idx] = value; +/* +* If the VCMDQ was already programmed (BASE) before mapping, fire +* setup on the ALLOC 0->1 transition; tear down on 1->0. +*/ +if (!was_alloc && now_alloc) { +tegra241_cmdqv_setup_vcmdq (cmdqv, idx, &local_err); + }else if (was_alloc && !now_alloc) { +tegra241_cmdqv_free_vcmdq (cmdqv, idx); + }break ; + } case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:tegra241_cmdqv_config_vintf_write (cmdqv, offset, value, &local_err);break ;@@ -427,12 +554,14 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ;tegra241_cmdqv_write_vcmdq_page1 (cmdqv, - offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false, + &local_err);break ; case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ;tegra241_cmdqv_write_vcmdq_page1 (cmdqv, - offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true, + &local_err);break ;default : qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",@@ -451,6 +580,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,static void tegra241_cmdqv_writell_mmio (Tegra241CMDQV *cmdqv,hwaddr offset,uint64_t value) { +Error *local_err =NULL ;int index;switch (offset) {@@ -463,18 +593,24 @@ static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, offset -=CMDQV_VINTF_PAGE1_BASE -CMDQV_VCMDQ_PAGE1_BASE ; index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ;tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, - offset - index *CMDQV_VCMDQ_STRIDE , index, value, false); + offset - index *CMDQV_VCMDQ_STRIDE , index, value, false, + &local_err);break ; case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H: index = (offset -CMDQV_VCMDQ_PAGE1_BASE ) /CMDQV_VCMDQ_STRIDE ;tegra241_cmdqv_write_vcmdq_page1_64 (cmdqv, - offset - index *CMDQV_VCMDQ_STRIDE , index, value, true); + offset - index *CMDQV_VCMDQ_STRIDE , index, value, true, + &local_err);break ;default : qemu_log_mask(LOG_UNIMP,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 616
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +4
-0"%s unhandled 64-bit write at 0x%" PRIx64" (WI)\n" , __func__, offset); } + +if (local_err) { +error_report_err (local_err); + } } static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
Otherwise looks good to me
Thanks
Eric
RE: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 109 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:37
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configuredExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 24 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
VCMDQ_ENlives inVCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO - so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
iommufd_backend_alloc_hw_queue()needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
On allocation failure, set
CMDQ_INIT_ERRand clearCMDQ_EN_OKin the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 11 +++ hw/arm/tegra241-cmdqv.c | 154+++++++++++++++++++++++++++++++++++++---
Quoted 36 lines
2 files changed, 156 insertions(+), 9 deletions(-)hw/arm/tegra241-cmdqv.h +11
-0@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */ @@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +0
-0hw/arm/tegra241-cmdqv.c line 21
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -18,6 +18,96 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, intindex)
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; +IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index]; + +if (!vcmdq) {hw/arm/tegra241-cmdqv.c
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+return ; + } +iommufd_backend_free_id (viommu->iommufd, vcmdq->hw_queue_id); +g_free (vcmdq); + cmdqv->vcmdq[index] =NULL ; +} + +static void tegra241_cmdqv_free_all_vcmdq (Tegra241CMDQV *cmdqv) +{ +/* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */
Quoted 5 lines
hw/arm/tegra241-cmdqv.c line 39
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+for (int i = (TEGRA241_CMDQV_MAX_CMDQ -1 ); i >=0 ; i--) { +tegra241_cmdqv_free_vcmdq (cmdqv, i); + }basically the same comment as previous patch. I would add in the exit phase reset at this stage too.
Yes. I get that now. Will populate the rest handler.
Quoted 40 lines
hw/arm/tegra241-cmdqv.c
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+} + +/* +* Allocate a host HW VCMDQ from the current cached BASE / size for @index.
hw/arm/tegra241-cmdqv.c
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9
-0+* +* Setup is a no-op (returns true) when any of the following preconditions +* isn't met yet: +* - BASE not prgrammed yet. +* - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0) +* - BASE / size don't resolve to a RAM region +* - CMDQV global enable or VINTF enable is not yet asserted +*/ +static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, intindex,
hw/arm/tegra241-cmdqv.c
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7
-0+ Error **errp) +{ +SMMUv3AccelState *accel = cmdqv->s_accel; +uint64_t base_mask = (uint64_t )R_VCMDQ0_BASE_L_ADDR_MASK | + (uint64_t )R_VCMDQ0_BASE_H_ADDR_MASK <<32 ; +uint64_t addr = cmdqv->vcmdq_base[index] & base_mask; + uint64_t log2 = cmdqv->vcmdq_base[index] &
R_VCMDQ0_BASE_L_LOG2SIZE_MASK;hw/arm/tegra241-cmdqv.c
Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12
-0+uint64_t size =1ULL << (log2 +4 ); +IOMMUFDViommu *viommu = accel->viommu; +IOMMUFDHWqueue *hw_queue; +uint32_t hw_queue_id; + +/* BASE not programmed yet. */ +if (!cmdqv->vcmdq_base[index]) { +return true; + } + +/* VCMDQ not yet mapped to a VINTF. */ + if (!(cmdqv->cmdq_alloc_map[index] &
R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {hw/arm/tegra241-cmdqv.c
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+return true; + } + + if (!tegra241_cmdqv_enabled(cmdqv) ||
Quoted 6 lines
!tegra241_vintf_enabled(cmdqv)) {
hw/arm/tegra241-cmdqv.c line 77
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+return true; + }nit: I think I would introduce an helper
vcmdq_hw_queue_ready_to_be_allocated(cmdqv, index)where you could put a proper doc comment.
Sure.
Quoted 5 lines
Not that once flag the base is set it returns true. ie. you do not check that both L & H are set.
But anyway the queue will be reallocated
That's true. I will capture it in comments.
Quoted 48 lines
hw/arm/tegra241-cmdqv.c
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +9
-0+ +tegra241_cmdqv_free_vcmdq (cmdqv, index); + +if (!iommufd_backend_alloc_hw_queue (viommu->iommufd, viommu- viommu_id, +IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV , + index, addr, size, &hw_queue_id, + errp)) { +/* Record the failure in the cache. */ + cmdqv->vcmdq_gerror[index] |=
R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;hw/arm/tegra241-cmdqv.c line 88
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+ cmdqv->vcmdq_status[index] &=~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
hw/arm/tegra241-cmdqv.c
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+return false; + } + hw_queue =g_new (IOMMUFDHWqueue,1 ); + hw_queue->hw_queue_id = hw_queue_id; + hw_queue->viommu = viommu; + cmdqv->vcmdq[index] = hw_queue; + + cmdqv->vcmdq_gerror[index] &=~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 241
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+ cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c line 108
Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12
-0+ +return true; +} + +static void tegra241_cmdqv_setup_all_vcmdq (Tegra241CMDQV *cmdqv, +Error **errp) +{ +for (int i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { +if (!tegra241_cmdqv_setup_vcmdq (cmdqv, i, errp)) { +return ; + } + }nit: you you could after the
tegra241_cmdqv_free_all_vcmdqhelperhw/arm/tegra241-cmdqv.c static void
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2
-0+} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. *
hw/arm/tegra241-cmdqv.c +0
-0@@ -145,7 +235,12 @@ static voidtegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +0
-1break ;case A_VCMDQ0_CONFIG :if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { - cmdqv->vcmdq_status[index] |=
Quoted 8 lines
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 243
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+/* Report init error if any. */ +if (!(cmdqv->vcmdq_gerror[index] & +R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK )) { + cmdqv->vcmdq_status[index] |= +R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }does it really belong to this patch?
I put it here because this is the patch that sets that error in the first place. Commit log kind of mentions it. I could explain it better in commit log if that's enough. The other option is splitting the error handling into a separate patch, but THB don't see a need for that.
Thanks,
Shameer
Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured
Quoted 224 lines · Shameer Kolothum Thodi
Quoted 108 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:37
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configuredExternal email: Use caution opening links or attachments
Hi Shameer,
Quoted 24 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.
VCMDQ_ENlives inVCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO - so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
iommufd_backend_alloc_hw_queue()needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.
On allocation failure, set
CMDQ_INIT_ERRand clearCMDQ_EN_OKin the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 11 +++ hw/arm/tegra241-cmdqv.c | 154+++++++++++++++++++++++++++++++++++++---
Quoted 36 lines
2 files changed, 156 insertions(+), 9 deletions(-)hw/arm/tegra241-cmdqv.h +11
-0@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {MemoryRegion mmio_cmdqv;qemu_irq irq;IOMMUFDVeventq *veventq; +IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0;/* CMDQ-V Config page register cache */ @@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1) SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1) +static inline bool tegra241_cmdqv_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->status &R_STATUS_CMDQV_ENABLED_MASK ; +} + +static inline bool tegra241_vintf_enabled (Tegra241CMDQV *cmdqv) +{ +return cmdqv->vintf_status &R_VINTF0_STATUS_ENABLE_OK_MASK ; +} +const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops (void );#endif /* HW_ARM_TEGRA241_CMDQV_H */ hw/arm/tegra241-cmdqv.c +0
-0hw/arm/tegra241-cmdqv.c line 21
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -18,6 +18,96 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, intindex)
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5
-0+{ +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; +IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index]; + +if (!vcmdq) {hw/arm/tegra241-cmdqv.c
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+return ; + } +iommufd_backend_free_id (viommu->iommufd, vcmdq->hw_queue_id); +g_free (vcmdq); + cmdqv->vcmdq[index] =NULL ; +} + +static void tegra241_cmdqv_free_all_vcmdq (Tegra241CMDQV *cmdqv) +{ +/* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */
Quoted 5 lines
hw/arm/tegra241-cmdqv.c line 39
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+for (int i = (TEGRA241_CMDQV_MAX_CMDQ -1 ); i >=0 ; i--) { +tegra241_cmdqv_free_vcmdq (cmdqv, i); + }basically the same comment as previous patch. I would add in the exit phase reset at this stage too.
Yes. I get that now. Will populate the rest handler.
Quoted 40 lines
hw/arm/tegra241-cmdqv.c
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+} + +/* +* Allocate a host HW VCMDQ from the current cached BASE / size for @index.
hw/arm/tegra241-cmdqv.c
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9
-0+* +* Setup is a no-op (returns true) when any of the following preconditions +* isn't met yet: +* - BASE not prgrammed yet. +* - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0) +* - BASE / size don't resolve to a RAM region +* - CMDQV global enable or VINTF enable is not yet asserted +*/ +static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, intindex,
hw/arm/tegra241-cmdqv.c
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7
-0+ Error **errp) +{ +SMMUv3AccelState *accel = cmdqv->s_accel; +uint64_t base_mask = (uint64_t )R_VCMDQ0_BASE_L_ADDR_MASK | + (uint64_t )R_VCMDQ0_BASE_H_ADDR_MASK <<32 ; +uint64_t addr = cmdqv->vcmdq_base[index] & base_mask; + uint64_t log2 = cmdqv->vcmdq_base[index] &
R_VCMDQ0_BASE_L_LOG2SIZE_MASK;hw/arm/tegra241-cmdqv.c
Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12
-0+uint64_t size =1ULL << (log2 +4 ); +IOMMUFDViommu *viommu = accel->viommu; +IOMMUFDHWqueue *hw_queue; +uint32_t hw_queue_id; + +/* BASE not programmed yet. */ +if (!cmdqv->vcmdq_base[index]) { +return true; + } + +/* VCMDQ not yet mapped to a VINTF. */ + if (!(cmdqv->cmdq_alloc_map[index] &
R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {hw/arm/tegra241-cmdqv.c
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+return true; + } + + if (!tegra241_cmdqv_enabled(cmdqv) ||Quoted 6 lines
!tegra241_vintf_enabled(cmdqv)) {
hw/arm/tegra241-cmdqv.c line 77
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2
-0+return true; + }nit: I think I would introduce an helper
vcmdq_hw_queue_ready_to_be_allocated(cmdqv, index)where you could put a proper doc comment.Sure.
Quoted 4 lines
Not that once flag the base is set it returns true. ie. you do not check that both L & H are set.
But anyway the queue will be reallocated
That's true. I will capture it in comments.
Quoted 55 lines
hw/arm/tegra241-cmdqv.c
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +9
-0+ +tegra241_cmdqv_free_vcmdq (cmdqv, index); + +if (!iommufd_backend_alloc_hw_queue (viommu->iommufd, viommu- viommu_id, +IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV , + index, addr, size, &hw_queue_id, + errp)) { +/* Record the failure in the cache. */ + cmdqv->vcmdq_gerror[index] |=
R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;hw/arm/tegra241-cmdqv.c line 88
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+ cmdqv->vcmdq_status[index] &=~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
hw/arm/tegra241-cmdqv.c
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+return false; + } + hw_queue =g_new (IOMMUFDHWqueue,1 ); + hw_queue->hw_queue_id = hw_queue_id; + hw_queue->viommu = viommu; + cmdqv->vcmdq[index] = hw_queue; + + cmdqv->vcmdq_gerror[index] &=~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 241
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+ cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c line 108
Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12
-0+ +return true; +} + +static void tegra241_cmdqv_setup_all_vcmdq (Tegra241CMDQV *cmdqv, +Error **errp) +{ +for (int i =0 ; i <TEGRA241_CMDQV_MAX_CMDQ ; i++) { +if (!tegra241_cmdqv_setup_vcmdq (cmdqv, i, errp)) { +return ; + } + }nit: you you could after the
tegra241_cmdqv_free_all_vcmdqhelperhw/arm/tegra241-cmdqv.c static void
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2
-0+} +/* * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets. *
hw/arm/tegra241-cmdqv.c +0
-0@@ -145,7 +235,12 @@ static voidtegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +0
-1break ;case A_VCMDQ0_CONFIG :if (value &R_VCMDQ0_CONFIG_CMDQ_EN_MASK ) { - cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+/* Report init error if any. */ +if (!(cmdqv->vcmdq_gerror[index] & +R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK )) { + cmdqv->vcmdq_status[index] |= +R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; + }
Quoted 5 lines
Quoted 1 line
does it really belong to this patch?
I put it here because this is the patch that sets that error in the first place. Commit log kind of mentions it. I could explain it better in commit log if that's enough. The other option is splitting the error handling into a separate patch, but THB don't see a need for that.
OK Thanks
Eric
Quoted 4 lines
Thanks,
Shameer
PATCH v6 19/31 hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
In reply to Shameer Kolothumhw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing
Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.
The two QEMU-trapped Page 0 apertures (direct at 0x10000, VINTF at 0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.
The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.
A VCMDQ Page 0 access is served from one of two sources:
- Cache-backed: no hw_queue is allocated for the VCMDQ (HW_QUEUE_ALLOC has not yet succeeded). Both apertures use QEMU's register cache.
- HW-backed: HW_QUEUE_ALLOC has succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.
tegra241_cmdqv_sync_vcmdq() copies any cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.
--- hw/arm/tegra241-cmdqv.c | 73 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+)
hw/arm/tegra241-cmdqv.c +73 -0
@@ -39,6 +39,47 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) } } +/* +* Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0 +* register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ +* has no hw_queue allocated or the VINTF page0 is not mmap'd. +*/ +static inline uint32_t *tegra241_cmdqv_vintf_ptr (Tegra241CMDQV *cmdqv, +int index,hwaddr offset0) +{ +if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) { +return NULL ; + } +return (uint32_t *)(cmdqv->vintf_page0 + + (index *CMDQV_VCMDQ_STRIDE ) + + (offset0 -CMDQV_VCMDQ_PAGE0_BASE )); +} + +/* +* Flush cached register writes into the mmap'd VINTF page0 after a +* successful HW_QUEUE_ALLOC, so the guest's earlier writes survive +* the cache-to-hardware transition. +*/ +static void tegra241_cmdqv_sync_vcmdq (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t *ptr; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONS_INDX ); +if (!ptr) { +return ; + } + *ptr = cmdqv->vcmdq_cons_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_PROD_INDX ); + *ptr = cmdqv->vcmdq_prod_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONFIG ); + *ptr = cmdqv->vcmdq_config[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_GERRORN ); + *ptr = cmdqv->vcmdq_gerrorn[index]; +} +/* * Allocate a host HW VCMDQ from the current cached BASE / size for @index. *
@@ -95,6 +136,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, cmdqv->vcmdq_gerror[index] &= ~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK ; cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +/* Pre-alloc cached writes survive the cache-to-hardware transition. */ +tegra241_cmdqv_sync_vcmdq (cmdqv, index); +return true; }
@@ -113,13 +157,22 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,* * The caller normalizes the MMIO offset such that @offset0 always refers * to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly +* from the VINTF page0 backing. Otherwise, fall back to the cache. */ static uint64_t tegra241_cmdqv_read_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index,bool direct) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0);uint64_t val =0 ; +if (ptr) { + val = *ptr; + goto out; + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : val = cmdqv->vcmdq_cons_indx[index];
@@ -144,6 +197,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_read_vcmdq_page0 (index, direct ?"direct" :"vi" , offset0, val);return val;
@@ -220,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,* * Page 0 registers are all 32-bit; this helper is only called for 4-byte * writes. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly +* to the VINTF page0 backing. Otherwise, update the cache. */ static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index,uint32_t value,bool direct) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0); + +if (ptr) { +switch (offset0) { +case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : cmdqv->vcmdq_cons_indx[index] = value;
@@ -255,6 +327,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_write_vcmdq_page0 (index, direct ?"direct" :"vi" , offset0, value); }
Signature · 1 line
2.43.0
Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) · line 48
Quoted 7 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +7
-0+/* +* Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0 +* register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ +* has no hw_queue allocated or the VINTF page0 is not mmap'd. +*/ +static inline uint32_t *tegra241_cmdqv_vintf_ptr (Tegra241CMDQV *cmdqv, +int index,hwaddr offset0)
I'd do:
tegra241_cmdqv_vintf_lvcmdq_ptr()
or just:
tegra241_vintf_lvcmdq_ptr()
Quoted 8 lines
static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
hwaddr offset0, int index,uint32_tvalue, bool direct)
{hw/arm/tegra241-cmdqv.c +4
-0+uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0); + +if (ptr) { +switch (offset0) {
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 298
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + }
Could stuff this into the helper? Return a valid ptr only when the offset0 is one of these four registers; otherwise, return NULL.
Then, here if would be:
if (ptr) {
*ptr = value;
goto out;
}Doing so inside the helper could also sanitize the other callers.
Otherwise, looks good to me.
RE: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 23 lines
-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 01 June 2026 23:06
To: Shameer Kolothum Thodi <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backinghw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 6 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +6
-0+/* +* Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0 +* register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ +* has no hw_queue allocated or the VINTF page0 is not mmap'd. +*/ +static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV*cmdqv,
Quoted 7 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ int index, hwaddr +offset0)I'd do:
tegra241_cmdqv_vintf_lvcmdq_ptr()
or just:
tegra241_vintf_lvcmdq_ptr()
Ok. I will go with tegra241_cmdqv_vintf_lvcmdq_ptr()
Quoted 20 lines
Quoted 1 line
static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV
*cmdqv,
Quoted 7 lines
hwaddr offset0, int index,
uint32_tvalue, bool
direct) {hw/arm/tegra241-cmdqv.c +4
-0+uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0); + +if (ptr) { +switch (offset0) {hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 298
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + }
Quoted 8 lines
Could stuff this into the helper? Return a valid ptr only when the offset0 is one of these four registers; otherwise, return NULL.
Then, here if would be:
if (ptr) { *ptr = value; goto out; }
The helper is shared with _read_vcmdq_page0(), where STATUS/ GERROR reads also need to route to the mmap when allocated. Folding the above would require a separate handling for "write" inside the helper. Could do. But I think this is more readable...
Thanks,
Shameer
Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
In reply to Shameer KolothumHi Shameer,
Quoted 1 line · Shameer Kolothum
hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing
the title is double here.
By the way may I suggest
hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 for mapped vcmdqs direct view
to emphasize this relates the direct view and only for mapped vcmdqs
Quoted 88 lines
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.The two QEMU-trapped Page 0 apertures (direct at
0x10000, VINTF at0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.
A VCMDQ Page 0 access is served from one of two sources:
- Cache-backed: no
hw_queueis allocated for the VCMDQ (HW_QUEUE_ALLOChas not yet succeeded). Both apertures use QEMU's register cache.- HW-backed:
HW_QUEUE_ALLOChas succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.
tegra241_cmdqv_sync_vcmdq()copies any cached writes (CONS_INDX,PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 73 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+)hw/arm/tegra241-cmdqv.c +41
-0@@ -39,6 +39,47 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) } } +/* +* Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0 +* register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ +* has no hw_queue allocated or the VINTF page0 is not mmap'd. +*/ +static inline uint32_t *tegra241_cmdqv_vintf_ptr (Tegra241CMDQV *cmdqv, +int index,hwaddr offset0) +{ +if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) { +return NULL ; + } +return (uint32_t *)(cmdqv->vintf_page0 + + (index *CMDQV_VCMDQ_STRIDE ) + + (offset0 -CMDQV_VCMDQ_PAGE0_BASE )); +} + +/* +* Flush cached register writes into the mmap'd VINTF page0 after a +* successful HW_QUEUE_ALLOC, so the guest's earlier writes survive +* the cache-to-hardware transition. +*/ +static void tegra241_cmdqv_sync_vcmdq (Tegra241CMDQV *cmdqv,int index) +{ +uint32_t *ptr; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONS_INDX ); +if (!ptr) { +return ; + } + *ptr = cmdqv->vcmdq_cons_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_PROD_INDX ); + *ptr = cmdqv->vcmdq_prod_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONFIG ); + *ptr = cmdqv->vcmdq_config[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_GERRORN ); + *ptr = cmdqv->vcmdq_gerrorn[index]; +} +/* * Allocate a host HW VCMDQ from the current cached BASE / size for @index. *
hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 139
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -95,6 +136,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, cmdqv->vcmdq_gerror[index] &= ~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK ; cmdqv->vcmdq_status[index] |=R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK ; +/* Pre-alloc cached writes survive the cache-to-hardware transition. */
what about the opposite transition?
Quoted 11 lines
hw/arm/tegra241-cmdqv.c +4
-0+tegra241_cmdqv_sync_vcmdq (cmdqv, index); +return true; }@@ -113,13 +157,22 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,* * The caller normalizes the MMIO offset such that @offset0 always refers * to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv, · line 171
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +3
-0+* from the VINTF page0 backing. Otherwise, fall back to the cache. */ static uint64_t tegra241_cmdqv_read_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index,bool direct) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0);uint64_t val =0 ; +if (ptr) {
shouldn't you test "direct" as well? I know that later on this is will be backed by another mr but I think this would be clearer.
Quoted 18 lines
hw/arm/tegra241-cmdqv.c +5
-0+ val = *ptr; + goto out; + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : val = cmdqv->vcmdq_cons_indx[index];@@ -144,6 +197,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_read_vcmdq_page0 (index, direct ?"direct" :"vi" , offset0, val);return val;@@ -220,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,* * Page 0 registers are all 32-bit; this helper is only called for 4-byte
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 285
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +4
-0* writes. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly +* to the VINTF page0 backing. Otherwise, update the cache. */ static void tegra241_cmdqv_write_vcmdq_page0 (Tegra241CMDQV *cmdqv,hwaddr offset0,int index,uint32_t value,bool direct) { +uint32_t *ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index, offset0);
same here?
Quoted 18 lines
hw/arm/tegra241-cmdqv.c +14
-0+ +if (ptr) { +switch (offset0) { +case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : cmdqv->vcmdq_cons_indx[index] = value;
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 331
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -255,6 +327,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset0); } +out: trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",
by the way could be interesting to log in the trace event that mmapped was used instead of cache regs.
besides this looks good to me
Thanks
Eric
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +0
-0offset0, value); }
RE: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 15 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:56
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 8 lines
External email: Use caution opening links or attachments
Hi Shameer,
Quoted 1 line · Shameer Kolothum
hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing
the title is double here.
Ah..Ok.
Quoted 5 lines
By the way may I suggest
hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 for mapped vcmdqs direct viewto emphasize this relates the direct view and only for mapped vcmdqs
It's true this eventually relates only to the direct aperture, but not yet. VINTF page0 only gets mapped into the guest in patch 21. Until then, for allocated vCMDQs we use the mmap'd VINTF page0 for both direct and VINTF aperture accesses.
May be we can rename to:
Route allocated VCMDQ Page0 accesses to the mmap'd VINTF page0
Quoted 96 lines
Quoted 26 lines
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.The two QEMU-trapped Page 0 apertures (direct at
0x10000, VINTF at0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.
A VCMDQ Page 0 access is served from one of two sources:
- Cache-backed: no
hw_queueis allocated for the VCMDQ (HW_QUEUE_ALLOChas not yet succeeded). Both apertures use QEMU's register cache.- HW-backed:
HW_QUEUE_ALLOChas succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.Quoted 7 lines
tegra241_cmdqv_sync_vcmdq()copies any cached writes (CONS_INDX,PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 73+++++++++++++++++++++++++++++++++++++++++
hw/arm/tegra241-cmdqv.c static void
Quoted 7 lines
1 file changed, 73 insertions(+)hw/arm/tegra241-cmdqv.c +0
-0@@ -39,6 +39,47 @@ static void
tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +6
-0} } +/* +* Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0 +* register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ +* has no hw_queue allocated or the VINTF page0 is not mmap'd. +*/ +static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV*cmdqv,
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+ int index, hwaddr offset0) +{ +if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) { +return NULL ; + } + return (uint32_t *)(cmdqv->vintf_page0 +hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ (index *CMDQV_VCMDQ_STRIDE ) + + (offset0 - CMDQV_VCMDQ_PAGE0_BASE)); +} + +/* +* Flush cached register writes into the mmap'd VINTF page0 after a +* successful HW_QUEUE_ALLOC, so the guest's earlier writes survive +* the cache-to-hardware transition. +*/ +static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, intindex)
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+{ +uint32_t *ptr; + + ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,
A_VCMDQ0_CONS_INDX);hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+if (!ptr) { +return ; + } + *ptr = cmdqv->vcmdq_cons_indx[index]; + + ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,
A_VCMDQ0_PROD_INDX);hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 11 lines
hw/arm/tegra241-cmdqv.c +9
-0+ *ptr = cmdqv->vcmdq_prod_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONFIG ); + *ptr = cmdqv->vcmdq_config[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_GERRORN ); + *ptr = cmdqv->vcmdq_gerrorn[index]; +} +/* * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
hw/arm/tegra241-cmdqv.c static bool
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +0
-0* @@ -95,6 +136,9 @@ static booltegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
Quoted 7 lines
hw/arm/tegra241-cmdqv.c static bool · line 136
Quoted 1 line
hw/arm/tegra241-cmdqv.c +0
-0cmdqv->vcmdq_gerror[index] &=~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
hw/arm/tegra241-cmdqv.c static bool · line 137
Quoted 1 line
hw/arm/tegra241-cmdqv.c +0
-0cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 139
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +1
-0+/* Pre-alloc cached writes survive the cache-to-hardware transition. */ what about the opposite transition?
Yeah. We don’t do anything now. I think better to reset the cache on opposite transition.
Quoted 22 lines
hw/arm/tegra241-cmdqv.c static void
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2
-0+tegra241_cmdqv_sync_vcmdq (cmdqv, index); +return true; }@@ -113,13 +157,22 @@ static voidtegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3
-0* * The caller normalizes the MMIO offset such that @offset0 always refers * to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly +* from the VINTF page0 backing. Otherwise, fall back to the cache. */ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV*cmdqv,
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +1
-0hwaddr offset0, int index, bool direct) { + uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);uint64_t val =0 ;
Quoted 3 lines
hw/arm/tegra241-cmdqv.c static void · line 171
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+if (ptr) {shouldn't you test "direct" as well? I know that later on this is will be backed by another mr but I think this would be clearer.
Is the suggestion here to just limit this to "direct"? We can't do that. Both apertures alias the same registers, and for an allocated and mmapped VINTF page0 we access the hw directly.
The vintf page0 gets mapped to the guest only in patch 21, so after that only the direct aperture is trapped here anyway.
Quoted 51 lines
hw/arm/tegra241-cmdqv.c static uint64_t
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +4
-0+ val = *ptr; + goto out; + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : val = cmdqv->vcmdq_cons_indx[index];@@ -144,6 +197,7 @@ static uint64_ttegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +1
-0"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_read_vcmdq_page0 (index, direct ?"direct" :"vi" , offset0, val);return val;@@ -220,11 +274,29 @@ static uint64_ttegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3
-0* * Page 0 registers are all 32-bit; this helper is only called for 4-byte * writes. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly +* to the VINTF page0 backing. Otherwise, update the cache. */ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV*cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 285
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0hwaddr offset0, int index, uint32_t value, bool direct) { + uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);same here?
hw/arm/tegra241-cmdqv.c static void
Quoted 18 lines
hw/arm/tegra241-cmdqv.c +14
-0+ +if (ptr) { +switch (offset0) { +case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : cmdqv->vcmdq_cons_indx[index] = value;@@ -255,6 +327,7 @@ static voidtegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
Quoted 7 lines
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 331
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1
-0"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset0); } +out: trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",by the way could be interesting to log in the trace event that mmapped was used instead of cache regs.
Could do.
Thanks,
Shameer
Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
Quoted 28 lines · Shameer Kolothum Thodi
Quoted 14 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:56
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backingQuoted 8 lines
External email: Use caution opening links or attachments
Hi Shameer,
Quoted 1 line · Shameer Kolothum
hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing
the title is double here.
Ah..Ok.
Quoted 4 lines
By the way may I suggest
hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 for mapped vcmdqs direct view
Quoted 8 lines
Quoted 1 line
to emphasize this relates the direct view and only for mapped vcmdqs
It's true this eventually relates only to the direct aperture, but not yet. VINTF page0 only gets mapped into the guest in patch 21. Until then, for allocated vCMDQs we use the mmap'd VINTF page0 for both direct and VINTF aperture accesses.
May be we can rename to:
Route allocated VCMDQ Page0 accesses to the mmap'd VINTF page0
Indeed. Yes above makes sense. It is about allocated VCMDQs only
Quoted 96 lines
Quoted 96 lines
Quoted 26 lines
Introduce
tegra241_cmdqv_vintf_ptr()to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.The two QEMU-trapped Page 0 apertures (direct at
0x10000, VINTF at0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.
A VCMDQ Page 0 access is served from one of two sources:
- Cache-backed: no
hw_queueis allocated for the VCMDQ (HW_QUEUE_ALLOChas not yet succeeded). Both apertures use QEMU's register cache.- HW-backed:
HW_QUEUE_ALLOChas succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.Quoted 7 lines
tegra241_cmdqv_sync_vcmdq()copies any cached writes (CONS_INDX,PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 73+++++++++++++++++++++++++++++++++++++++++
hw/arm/tegra241-cmdqv.c static void
Quoted 7 lines
1 file changed, 73 insertions(+)hw/arm/tegra241-cmdqv.c +0
-0@@ -39,6 +39,47 @@ static void
tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +6
-0} } +/* +* Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0 +* register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ +* has no hw_queue allocated or the VINTF page0 is not mmap'd. +*/ +static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV*cmdqv,
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+ int index, hwaddr offset0) +{ +if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) { +return NULL ; + } + return (uint32_t *)(cmdqv->vintf_page0 +hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+ (index *CMDQV_VCMDQ_STRIDE ) + + (offset0 - CMDQV_VCMDQ_PAGE0_BASE)); +} + +/* +* Flush cached register writes into the mmap'd VINTF page0 after a +* successful HW_QUEUE_ALLOC, so the guest's earlier writes survive +* the cache-to-hardware transition. +*/ +static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, intindex)
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+{ +uint32_t *ptr; + + ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,
A_VCMDQ0_CONS_INDX);hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+if (!ptr) { +return ; + } + *ptr = cmdqv->vcmdq_cons_indx[index]; + + ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,
A_VCMDQ0_PROD_INDX);hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
Quoted 11 lines
hw/arm/tegra241-cmdqv.c +9
-0+ *ptr = cmdqv->vcmdq_prod_indx[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_CONFIG ); + *ptr = cmdqv->vcmdq_config[index]; + + ptr =tegra241_cmdqv_vintf_ptr (cmdqv, index,A_VCMDQ0_GERRORN ); + *ptr = cmdqv->vcmdq_gerrorn[index]; +} +/* * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
hw/arm/tegra241-cmdqv.c static bool
Quoted 2 lines
hw/arm/tegra241-cmdqv.c +0
-0* @@ -95,6 +136,9 @@ static booltegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
Quoted 8 lines
Quoted 6 lines
hw/arm/tegra241-cmdqv.c static bool · line 136
Quoted 1 line
hw/arm/tegra241-cmdqv.c +0
-0cmdqv->vcmdq_gerror[index] &=~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
hw/arm/tegra241-cmdqv.c static bool · line 137
Quoted 1 line
hw/arm/tegra241-cmdqv.c +0
-0cmdqv->vcmdq_status[index] |=
R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 139
Quoted 1 line
hw/arm/tegra241-cmdqv.c +1
-0+/* Pre-alloc cached writes survive the cache-to-hardware transition. */ what about the opposite transition?
Yeah. We don’t do anything now. I think better to reset the cache on opposite transition.
OK. I would add this explanation.
Quoted 24 lines
Quoted 24 lines
hw/arm/tegra241-cmdqv.c static void
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +2
-0+tegra241_cmdqv_sync_vcmdq (cmdqv, index); +return true; }@@ -113,13 +157,22 @@ static voidtegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3
-0* * The caller normalizes the MMIO offset such that @offset0 always refers * to a VCMDQ0_* register, while @index selects the VCMDQ instance. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly +* from the VINTF page0 backing. Otherwise, fall back to the cache. */ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV*cmdqv,
Quoted 7 lines
hw/arm/tegra241-cmdqv.c +2
-0hwaddr offset0, int index, bool direct) { + uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);uint64_t val =0 ; +if (ptr) {
Quoted 8 lines
Quoted 2 lines
shouldn't you test "direct" as well? I know that later on this is will be backed by another mr but I think this would be clearer.
Is the suggestion here to just limit this to "direct"? We can't do that. Both apertures alias the same registers, and for an allocated and mmapped VINTF page0 we access the hw directly.
The vintf page0 gets mapped to the guest only in patch 21, so after that only the direct aperture is trapped here anyway.
Yes I agree this would be true only once patch 21 is applied. Sorry for the noise.
Quoted 57 lines
Quoted 57 lines
hw/arm/tegra241-cmdqv.c static uint64_t
Quoted 9 lines
hw/arm/tegra241-cmdqv.c +4
-0+ val = *ptr; + goto out; + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : val = cmdqv->vcmdq_cons_indx[index];@@ -144,6 +197,7 @@ static uint64_ttegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +1
-0"%s unhandled read access at 0x%" PRIx64"\n" , __func__, offset0); } +out:trace_tegra241_cmdqv_read_vcmdq_page0 (index, direct ?"direct" :"vi" , offset0, val);return val;@@ -220,11 +274,29 @@ static uint64_ttegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3
-0* * Page 0 registers are all 32-bit; this helper is only called for 4-byte * writes. +* +* If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly +* to the VINTF page0 backing. Otherwise, update the cache. */ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV*cmdqv,
hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 285
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0hwaddr offset0, int index, uint32_t value, bool direct) { + uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);same here?
hw/arm/tegra241-cmdqv.c static void
Quoted 18 lines
hw/arm/tegra241-cmdqv.c +14
-0+ +if (ptr) { +switch (offset0) { +case A_VCMDQ0_CONS_INDX : +case A_VCMDQ0_PROD_INDX : +case A_VCMDQ0_CONFIG : +case A_VCMDQ0_GERRORN : + *ptr = value; + goto out; +default : +break ; + } + } +switch (offset0) {case A_VCMDQ0_CONS_INDX : cmdqv->vcmdq_cons_indx[index] = value;@@ -255,6 +327,7 @@ static voidtegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1
-0"%s unhandled write access at 0x%" PRIx64"\n" , __func__, offset0); } +out: trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",
Quoted 6 lines
Quoted 2 lines
by the way could be interesting to log in the trace event that mmapped was used instead of cache regs.
Could do.
Thanks,
Shameer
Thanks
Eric
PATCH v6 20/31 memory: Allow RAM device regions to skip IOMMU mapping
In reply to Shameer KolothumSome RAM device regions created with memory_region_init_ram_device_ptr() are not intended to be P2P DMA targets.
The VFIO listener currently treats all RAM device regions as DMA capable and attempts to map them into the IOMMU. For regions without dma-buf backing this fails and prints warnings such as:
IOMMU_IOAS_MAP failed: Bad address, PCI BAR?
Introduce a MemoryRegion flag (ram_device_skip_iommu_map) to mark RAM device regions that should not be IOMMU mapped, paired with memory_region_skip_iommu_map() / memory_region_set_skip_iommu_map() accessors. When the flag is set, the VFIO listener skips DMA mapping for that region.
--- include/system/memory.h | 21 +++++++++++++++++++++ hw/vfio/listener.c | 6 ++++++ system/memory.c | 10 ++++++++++ hw/vfio/trace-events | 1 + 4 files changed, 38 insertions(+)
include/system/memory.h +21 -0
@@ -864,6 +864,8 @@ struct MemoryRegion {/* For devices designed to perform re-entrant IO into their own IO MRs */ bool disable_reentrancy_guard; +/* RAM device region that does not require IOMMU mapping for P2P */ +bool ram_device_skip_iommu_map; };struct IOMMUMemoryRegion {
@@ -1743,6 +1745,25 @@ static inline bool memory_region_is_romd(const MemoryRegion *mr)*/ bool memory_region_is_protected (const MemoryRegion *mr); +/** +* memory_region_skip_iommu_map: check whether a memory region is excluded +* from IOMMU mapping +* +* Returns %true if @mr is a RAM device region marked to skip IOMMU mapping. +* +* @mr: the memory region being queried +*/ +bool memory_region_skip_iommu_map (const MemoryRegion *mr); + +/** +* memory_region_set_skip_iommu_map: mark a RAM device region to skip IOMMU +* mapping +* +* @mr: the memory region being modified +* @skip: %true to skip IOMMU mapping, %false to allow it +*/ +void memory_region_set_skip_iommu_map (MemoryRegion *mr,bool skip); +/** * memory_region_has_guest_memfd: check whether a memory region has guest_memfd * associated
hw/vfio/listener.c +6 -0
@@ -610,6 +610,12 @@ void vfio_container_region_add(VFIOContainer *bcontainer, } } +if (memory_region_skip_iommu_map (section->mr)) { +trace_vfio_listener_region_skip_dma_map (memory_region_name (section->mr), + iova,int128_get64 (llsize)); +return ; + } + ret =vfio_container_dma_map (bcontainer, iova,int128_get64 (llsize), vaddr, section->readonly, section->mr);if (ret) {
system/memory.c +10 -0
@@ -1814,6 +1814,16 @@ bool memory_region_is_protected(const MemoryRegion *mr)return mr->ram && (mr->ram_block->flags &RAM_PROTECTED ); } +bool memory_region_skip_iommu_map (const MemoryRegion *mr) +{ +return memory_region_is_ram_device (mr) && mr->ram_device_skip_iommu_map; +} + +void memory_region_set_skip_iommu_map (MemoryRegion *mr,bool skip) +{ + mr->ram_device_skip_iommu_map = skip; +} +bool memory_region_has_guest_memfd (const MemoryRegion *mr) {return mr->ram_block && mr->ram_block->guest_memfd >=0 ;
hw/vfio/trace-events +1 -0
@@ -100,6 +100,7 @@ vfio_listener_region_del_iommu(const char *name) "region_del [iommu] %s" vfio_listener_region_add_ram(uint64_t iova_start, uint64_t iova_end, void *vaddr) "region_add [ram] 0x%"PRIx64" - 0x%"PRIx64" [%p]" vfio_known_safe_misalignment(const char *name, uint64_t iova, uint64_t offset_within_region, uintptr_t page_size) "Region \"%s\" iova=0x%"PRIx64" offset_within_region=0x%"PRIx64" qemu_real_host_page_size=0x%"PRIxPTR vfio_listener_region_add_no_dma_map(const char *name, uint64_t iova, uint64_t size, uint64_t page_size) "Region \"%s\" 0x%"PRIx64" size=0x%"PRIx64" is not aligned to 0x%"PRIx64" and cannot be mapped for DMA" +vfio_listener_region_skip_dma_map(const char *name, uint64_t iova, uint64_t size) "Region \"%s\" 0x%"PRIx64" size=0x%"PRIx64" marked to skip IOMMU mapping" vfio_listener_region_del(uint64_t start, uint64_t end) "region_del 0x%"PRIx64" - 0x%"PRIx64 vfio_device_dirty_tracking_update(uint64_t start, uint64_t end, uint64_t min, uint64_t max) "section 0x%"PRIx64" - 0x%"PRIx64" -> update [0x%"PRIx64" - 0x%"PRIx64"]" vfio_device_dirty_tracking_start(int nr_ranges, uint64_t min32, uint64_t max32, uint64_t min64, uint64_t max64, uint64_t minpci, uint64_t maxpci) "nr_ranges %d 32:[0x%"PRIx64" - 0x%"PRIx64"], 64:[0x%"PRIx64" - 0x%"PRIx64"], pci64:[0x%"PRIx64" - 0x%"PRIx64"]"
Signature · 1 line
2.43.0
PATCH v6 21/31 hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Install the mmap'd vintf_page0 region as a RAM-device MemoryRegion at guest MMIO offset 0x30000 (the VINTF Page 0 aperture) when VINTF is enabled, and remove it on VINTF disable. This eliminates QEMU trapping for hot-path CONS/PROD index updates via the VINTF aperture.
After this patch, the two VCMDQ Page 0 apertures use different access paths: the direct aperture (0x10000) remains QEMU-trapped, while the VINTF aperture (0x30000) is a guest-direct RAM mapping.
The direct aperture is intentionally kept trapped (not aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing would route those writes to unallocated logical slots in the VINTF page, where the hardware silently drops them.
--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+)
hw/arm/tegra241-cmdqv.h +1 -0
@@ -49,6 +49,7 @@ typedef struct Tegra241CMDQV {IOMMUFDVeventq *veventq;IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0; +MemoryRegion *mr_vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;
hw/arm/tegra241-cmdqv.c +36 -0
@@ -18,6 +18,40 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_guest_unmap_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +if (!cmdqv->mr_vintf_page0) { +return ; + } + +memory_region_del_subregion (&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0); +object_unparent (OBJECT (cmdqv->mr_vintf_page0)); +g_free (cmdqv->mr_vintf_page0); + cmdqv->mr_vintf_page0 =NULL ; +} + +static void tegra241_cmdqv_guest_map_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +char *name; + +if (cmdqv->mr_vintf_page0) { +return ; + } + + name =g_strdup_printf ("%s vintf-page0" , +memory_region_name (&cmdqv->mmio_cmdqv)); + cmdqv->mr_vintf_page0 =g_malloc0 (sizeof (*cmdqv->mr_vintf_page0)); +memory_region_init_ram_device_ptr (cmdqv->mr_vintf_page0, +memory_region_owner (&cmdqv->mmio_cmdqv), + name,VINTF_PAGE_SIZE , + cmdqv->vintf_page0); +memory_region_set_skip_iommu_map (cmdqv->mr_vintf_page0, true); +memory_region_add_subregion_overlap (&cmdqv->mmio_cmdqv, +CMDQV_VINTF_PAGE0_BASE , + cmdqv->mr_vintf_page0,1 ); +g_free (name); +} +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) {IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
@@ -445,6 +479,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,if (value &R_VINTF0_CONFIG_ENABLE_MASK ) {if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +tegra241_cmdqv_guest_map_vintf_page0 (cmdqv);/* * VCMDQs whose BASE was programmed before VINTF was * enabled need their hw_queue allocated now.
@@ -452,6 +487,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,tegra241_cmdqv_setup_all_vcmdq (cmdqv, errp); } }else { +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv);tegra241_cmdqv_free_all_vcmdq (cmdqv);tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ;
Signature · 1 line
2.43.0
Re: [PATCH v6 21/31] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space
Hi Shammer,
Quoted 1 line · Shameer Kolothum
From: Nicolin Chen <[email protected]>
I would disambiguate the title a little bit:
Use mmap'd host VINTF page0 for virtual VINTF page0
Also in previous patch titles you could emphase the mmap'd page is the host one in a similar way.
Quoted 83 lines
Install the mmap'd
vintf_page0region as a RAM-device MemoryRegion at guest MMIO offset0x30000(the VINTF Page 0 aperture) when VINTF is enabled, and remove it on VINTF disable. This eliminates QEMU trapping for hot-path CONS/PROD index updates via the VINTF aperture.After this patch, the two VCMDQ Page 0 apertures use different access paths: the direct aperture (
0x10000) remains QEMU-trapped, while the VINTF aperture (0x30000) is a guest-direct RAM mapping.The direct aperture is intentionally kept trapped (not aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing would route those writes to unallocated logical slots in the VINTF page, where the hardware silently drops them.
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+)hw/arm/tegra241-cmdqv.h +1
-0@@ -49,6 +49,7 @@ typedef struct Tegra241CMDQV {IOMMUFDVeventq *veventq;IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0; +MemoryRegion *mr_vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c +34
-0@@ -18,6 +18,40 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_guest_unmap_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +if (!cmdqv->mr_vintf_page0) { +return ; + } + +memory_region_del_subregion (&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0); +object_unparent (OBJECT (cmdqv->mr_vintf_page0)); +g_free (cmdqv->mr_vintf_page0); + cmdqv->mr_vintf_page0 =NULL ; +} + +static void tegra241_cmdqv_guest_map_vintf_page0 (Tegra241CMDQV *cmdqv) +{ +char *name; + +if (cmdqv->mr_vintf_page0) { +return ; + } + + name =g_strdup_printf ("%s vintf-page0" , +memory_region_name (&cmdqv->mmio_cmdqv)); + cmdqv->mr_vintf_page0 =g_malloc0 (sizeof (*cmdqv->mr_vintf_page0)); +memory_region_init_ram_device_ptr (cmdqv->mr_vintf_page0, +memory_region_owner (&cmdqv->mmio_cmdqv), + name,VINTF_PAGE_SIZE , + cmdqv->vintf_page0); +memory_region_set_skip_iommu_map (cmdqv->mr_vintf_page0, true); +memory_region_add_subregion_overlap (&cmdqv->mmio_cmdqv, +CMDQV_VINTF_PAGE0_BASE , + cmdqv->mr_vintf_page0,1 ); +g_free (name); +} +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) {IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 482
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1
-0@@ -445,6 +479,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,if (value &R_VINTF0_CONFIG_ENABLE_MASK ) {if (tegra241_cmdqv_mmap_vintf_page0 (cmdqv, errp)) { cmdqv->vintf_status |=R_VINTF0_STATUS_ENABLE_OK_MASK ; +tegra241_cmdqv_guest_map_vintf_page0 (cmdqv);
don't you want to do that only after the tegra241_cmdqv_setup_all_vcmdq()?
hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 493
Quoted 11 lines
hw/arm/tegra241-cmdqv.c +1
-0/* * VCMDQs whose BASE was programmed before VINTF was * enabled need their hw_queue allocated now. @@ -452,6 +487,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,tegra241_cmdqv_setup_all_vcmdq (cmdqv, errp); } }else { +tegra241_cmdqv_guest_unmap_vintf_page0 (cmdqv);tegra241_cmdqv_free_all_vcmdq (cmdqv);tegra241_cmdqv_munmap_vintf_page0 (cmdqv, errp); cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK ;
Thanks
Eric
RE: [PATCH v6 21/31] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space
Quoted 20 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 11:07
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 21/31] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO spaceExternal email: Use caution opening links or attachments
Hi Shammer,
Quoted 7 lines
Quoted 1 line · Shameer Kolothum
From: Nicolin Chen <[email protected]>
I would disambiguate the title a little bit:
Use mmap'd host VINTF page0 for virtual VINTF page0Also in previous patch titles you could emphase the mmap'd page is the host one in a similar way.
Ok.
Quoted 84 lines
Quoted 16 lines
Install the mmap'd
vintf_page0region as a RAM-device MemoryRegion at guest MMIO offset0x30000(the VINTF Page 0 aperture) when VINTF is enabled, and remove it on VINTF disable. This eliminates QEMU trapping for hot-path CONS/PROD index updates via the VINTF aperture.After this patch, the two VCMDQ Page 0 apertures use different access
paths: the direct aperture (0x10000) remains QEMU-trapped, while the VINTF aperture (0x30000) is a guest-direct RAM mapping.The direct aperture is intentionally kept trapped (not aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing would route those writes to unallocated logical slots in the VINTF page, where the hardware silently drops them.
Quoted 6 lines
Signed-off-by Nicolin Chen <[email protected]>Co-developed-by Shameer Kolothum <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.h | 1 + hw/arm/tegra241-cmdqv.c | 36++++++++++++++++++++++++++++++++++++
Quoted 59 lines
2 files changed, 37 insertions(+)hw/arm/tegra241-cmdqv.h index renamed from hw/arm/tegra241-cmdqv.h +0
-0
84499b840d..01cd6af97d100644hw/arm/tegra241-cmdqv.h +1
-0@@ -49,6 +49,7 @@ typedef struct Tegra241CMDQV {IOMMUFDVeventq *veventq;IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ ];void *vintf_page0; +MemoryRegion *mr_vintf_page0;/* CMDQ-V Config page register cache */ uint32_t config;hw/arm/tegra241-cmdqv.c index renamed from hw/arm/tegra241-cmdqv.c +0
-0
dce7089697..cdd4fa4f1a100644hw/arm/tegra241-cmdqv.c +34
-0@@ -18,6 +18,40 @@#include "tegra241-cmdqv.h" #include "trace.h" +static void tegra241_cmdqv_guest_unmap_vintf_page0 (Tegra241CMDQV +*cmdqv) { +if (!cmdqv->mr_vintf_page0) { +return ; + } + +memory_region_del_subregion (&cmdqv->mmio_cmdqv, cmdqv- mr_vintf_page0); +object_unparent (OBJECT (cmdqv->mr_vintf_page0)); +g_free (cmdqv->mr_vintf_page0); + cmdqv->mr_vintf_page0 =NULL ; +} + +static void tegra241_cmdqv_guest_map_vintf_page0 (Tegra241CMDQV +*cmdqv) { +char *name; + +if (cmdqv->mr_vintf_page0) { +return ; + } + + name =g_strdup_printf ("%s vintf-page0" , +memory_region_name (&cmdqv->mmio_cmdqv)); + cmdqv->mr_vintf_page0 =g_malloc0 (sizeof (*cmdqv->mr_vintf_page0)); +memory_region_init_ram_device_ptr (cmdqv->mr_vintf_page0, +memory_region_owner (&cmdqv->mmio_cmdqv), + name,VINTF_PAGE_SIZE , + cmdqv->vintf_page0); +memory_region_set_skip_iommu_map (cmdqv->mr_vintf_page0, true); +memory_region_add_subregion_overlap (&cmdqv->mmio_cmdqv, +CMDQV_VINTF_PAGE0_BASE , + cmdqv->mr_vintf_page0,1 ); +g_free (name); +} +static void tegra241_cmdqv_free_vcmdq (Tegra241CMDQV *cmdqv,int index) {Quoted 1 line
IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; @@ -445,6+479,7
Quoted 8 lines
Quoted 1 line
@@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV
*cmdqv,
Quoted 5 lines
if (value & R_VINTF0_CONFIG_ENABLE_MASK) { if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) { cmdqv->vintf_status |=
R_VINTF0_STATUS_ENABLE_OK_MASK;
+tegra241_cmdqv_guest_map_vintf_page0(cmdqv);don't you want to do that only after the
tegra241_cmdqv_setup_all_vcmdq()?
Functionally same. Could do, reads better, I think.
Thanks,
Shameer
PATCH v6 22/31 hw/arm/smmuv3-accel: Introduce common helper for veventq read
In reply to Shameer KolothumMove the vEVENTQ read and validation logic into a common helper smmuv3_accel_event_read_validate(). The helper performs the read(), checks for overflow and short reads, validates the sequence number, and updates the sequence state.
This helper can be reused for Tegra241 CMDQV vEVENTQ support in a subsequent patch.
Error handling is slightly adjusted: instead of reporting errors directly in the read handler, the helper now returns errors via Error **. Sequence gaps are reported as warnings.
--- hw/arm/smmuv3-accel.h | 2 ++ hw/arm/smmuv3-accel-stubs.c | 7 ++++ hw/arm/smmuv3-accel.c | 67 ++++++++++++++++++++++--------------- 3 files changed, 49 insertions(+), 27 deletions(-)
hw/arm/smmuv3-accel.h +2 -0
@@ -87,6 +87,8 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *s, void *cmd, SMMUDevice *sdev, Error **errp);void smmuv3_accel_idr_override (SMMUv3State *s);bool smmuv3_accel_alloc_veventq (SMMUv3State *s,Error **errp); +bool smmuv3_accel_event_read_validate (IOMMUFDVeventq *veventq,uint32_t type, +void *buf,size_t size,Error **errp);void smmuv3_accel_reset (SMMUv3State *s);#endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +7 -0
@@ -47,6 +47,13 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)return true; } +bool smmuv3_accel_event_read_validate (IOMMUFDVeventq *veventq,uint32_t type, +void *buf,size_t size,Error **errp) +{ +return true; +} + +void smmuv3_accel_reset (SMMUv3State *s) { }
hw/arm/smmuv3-accel.c +40 -27
87 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v6 23/31 hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
In reply to Shameer KolothumInstall an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.
The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().
--- hw/arm/tegra241-cmdqv.c | 63 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 64 insertions(+)
hw/arm/tegra241-cmdqv.c +63 -0
@@ -14,6 +14,7 @@#include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" +#include "hw/core/irq.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" #include "trace.h"
@@ -751,6 +752,51 @@ out:trace_tegra241_cmdqv_write_mmio (offset, value, size); } +static void tegra241_cmdqv_event_read (void *opaque) +{ +Tegra241CMDQV *cmdqv = opaque; +IOMMUFDVeventq *veventq = cmdqv->veventq; +struct { +struct iommufd_vevent_header hdr; +struct iommu_vevent_tegra241_cmdqv vevent; + } buf; +Error *local_err =NULL ; + +if (!smmuv3_accel_event_read_validate (veventq, +IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV , + &buf,sizeof (buf), &local_err)) { +warn_report_err_once (local_err); +return ; + } + +if (buf.vevent.lvcmdq_err_map[0 ] || buf.vevent.lvcmdq_err_map[1 ]) { + cmdqv->vintf_cmdq_err_map[0 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[1 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],32 ,32 ); + cmdqv->vintf_cmdq_err_map[2 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[3 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],32 ,32 ); +/* +* CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct +* registers (different MMIO offsets). With only VINTF0 exposed +* they carry the same data, so mirror. +*/ +for (int i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i]; + } +/* Set the VINTF0 bit in VI_ERR_MAP_0 (only VINTF0 is exposed). */ + cmdqv->vi_err_map[0 ] |=BIT (0 ); +if (!(cmdqv->vi_int_mask[0 ] &BIT (0 ))) { +qemu_irq_pulse (cmdqv->irq); + } +trace_tegra241_cmdqv_err_map ( + cmdqv->vintf_cmdq_err_map[3 ], cmdqv->vintf_cmdq_err_map[2 ], + cmdqv->vintf_cmdq_err_map[1 ], cmdqv->vintf_cmdq_err_map[0 ]); + } +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) {SMMUv3AccelState *accel = s->s_accel;
@@ -762,6 +808,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)return ; }if (veventq) { +qemu_set_fd_handler (veventq->veventq_fd,NULL ,NULL ,NULL );close (veventq->veventq_fd);iommufd_backend_free_id (viommu->iommufd, veventq->veventq_id);g_free (veventq);
@@ -777,6 +824,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;uint32_t viommu_id, veventq_id, veventq_fd;IOMMUFDVeventq *veventq; +int flags; if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid, IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
@@ -795,14 +843,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, goto free_viommu; } + flags =fcntl (veventq_fd,F_GETFL ); +if (flags <0 ) { +error_setg (errp,"Failed to get flags for vEVENTQ fd" ); + goto free_veventq; + } +if (fcntl (veventq_fd,F_SETFL ,O_NONBLOCK | flags) <0 ) { +error_setg (errp,"Failed to set O_NONBLOCK on vEVENTQ fd" ); + goto free_veventq; + } + veventq =g_new (IOMMUFDVeventq,1 ); veventq->veventq_id = veventq_id; veventq->veventq_fd = veventq_fd; cmdqv->veventq = veventq; +/* Set up event handler for veventq fd */ +qemu_set_fd_handler (veventq_fd, tegra241_cmdqv_event_read,NULL , cmdqv); *out_viommu_id = viommu_id;return true; +free_veventq: +close (veventq_fd); +iommufd_backend_free_id (idev->iommufd, veventq_id); free_viommu:iommufd_backend_free_id (idev->iommufd, viommu_id);return false;
hw/arm/trace-events +1 -0
@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
Re: [PATCH v6 23/31] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
Quoted 8 lines · Shameer Kolothum
Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.
The handler runs in QEMU's main loop, using a non-blocking fd registered via
qemu_set_fd_handler().Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Eric
Quoted 127 lines
--- hw/arm/tegra241-cmdqv.c | 63 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 64 insertions(+)hw/arm/tegra241-cmdqv.c +63
-0@@ -14,6 +14,7 @@#include "hw/arm/smmuv3.h" #include "hw/arm/smmuv3-common.h" +#include "hw/core/irq.h" #include "smmuv3-accel.h" #include "tegra241-cmdqv.h" #include "trace.h" @@ -751,6 +752,51 @@ out:trace_tegra241_cmdqv_write_mmio (offset, value, size); } +static void tegra241_cmdqv_event_read (void *opaque) +{ +Tegra241CMDQV *cmdqv = opaque; +IOMMUFDVeventq *veventq = cmdqv->veventq; +struct { +struct iommufd_vevent_header hdr; +struct iommu_vevent_tegra241_cmdqv vevent; + } buf; +Error *local_err =NULL ; + +if (!smmuv3_accel_event_read_validate (veventq, +IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV , + &buf,sizeof (buf), &local_err)) { +warn_report_err_once (local_err); +return ; + } + +if (buf.vevent.lvcmdq_err_map[0 ] || buf.vevent.lvcmdq_err_map[1 ]) { + cmdqv->vintf_cmdq_err_map[0 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[1 ] = +extract64 (buf.vevent.lvcmdq_err_map[0 ],32 ,32 ); + cmdqv->vintf_cmdq_err_map[2 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],0 ,32 ); + cmdqv->vintf_cmdq_err_map[3 ] = +extract64 (buf.vevent.lvcmdq_err_map[1 ],32 ,32 ); +/* +* CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct +* registers (different MMIO offsets). With only VINTF0 exposed +* they carry the same data, so mirror. +*/ +for (int i =0 ; i <4 ; i++) { + cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i]; + } +/* Set the VINTF0 bit in VI_ERR_MAP_0 (only VINTF0 is exposed). */ + cmdqv->vi_err_map[0 ] |=BIT (0 ); +if (!(cmdqv->vi_int_mask[0 ] &BIT (0 ))) { +qemu_irq_pulse (cmdqv->irq); + } +trace_tegra241_cmdqv_err_map ( + cmdqv->vintf_cmdq_err_map[3 ], cmdqv->vintf_cmdq_err_map[2 ], + cmdqv->vintf_cmdq_err_map[1 ], cmdqv->vintf_cmdq_err_map[0 ]); + } +} +static void tegra241_cmdqv_free_viommu (SMMUv3State *s) {SMMUv3AccelState *accel = s->s_accel;@@ -762,6 +808,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)return ; }if (veventq) { +qemu_set_fd_handler (veventq->veventq_fd,NULL ,NULL ,NULL );close (veventq->veventq_fd);iommufd_backend_free_id (viommu->iommufd, veventq->veventq_id);g_free (veventq);@@ -777,6 +824,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;uint32_t viommu_id, veventq_id, veventq_fd;IOMMUFDVeventq *veventq; +int flags; if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid, IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,@@ -795,14 +843,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, goto free_viommu; } + flags =fcntl (veventq_fd,F_GETFL ); +if (flags <0 ) { +error_setg (errp,"Failed to get flags for vEVENTQ fd" ); + goto free_veventq; + } +if (fcntl (veventq_fd,F_SETFL ,O_NONBLOCK | flags) <0 ) { +error_setg (errp,"Failed to set O_NONBLOCK on vEVENTQ fd" ); + goto free_veventq; + } + veventq =g_new (IOMMUFDVeventq,1 ); veventq->veventq_id = veventq_id; veventq->veventq_fd = veventq_fd; cmdqv->veventq = veventq; +/* Set up event handler for veventq fd */ +qemu_set_fd_handler (veventq_fd, tegra241_cmdqv_event_read,NULL , cmdqv); *out_viommu_id = viommu_id;return true; +free_veventq: +close (veventq_fd); +iommufd_backend_free_id (idev->iommufd, veventq_id); free_viommu:iommufd_backend_free_id (idev->iommufd, viommu_id);return false;hw/arm/trace-events +1
-0@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS # tegra241-cmdqv tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" +tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
PATCH v6 24/31 hw/arm/tegra241-cmdqv: Add reset handler
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.
--- hw/arm/tegra241-cmdqv.h | 3 +++ hw/arm/tegra241-cmdqv.c | 48 +++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 3 files changed, 52 insertions(+)
hw/arm/tegra241-cmdqv.h +3 -0
@@ -89,6 +89,9 @@ FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8) FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)FIELD (CONFIG ,CONS_DRAM_EN ,20 ,1 ) +/* CMDQV_EN=1, PER_CMD_OFFSET=16B, CLK_BATCH=256, CMD_BATCH=32. */ +#define V_CONFIG_RESET 0x00020083 + REG32(PARAM, 0x4) FIELD(PARAM, CMDQV_VER, 0, 4)FIELD (PARAM ,CMDQV_NUM_CMDQ_LOG2 ,4 ,4 )
hw/arm/tegra241-cmdqv.c +48 -0
56 lines not on this page · raw message
hw/arm/trace-events +1 -0
@@ -76,6 +76,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x" tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X" +tegra241_cmdqv_init_regs(uint32_t param) "register init, param=0x%08X" tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
PATCH v6 25/31 hw/arm/tegra241-cmdqv: Limit queue size based on backend page size
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
CMDQV HW performs DMA accesses to guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.
Use qemu_minrampagesize() to find the smallest memory-backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.
Note this is done at SMMUv3 init, before any guest queue GPA is known, so the cap is conservative. Maximum queue size is 8MiB; it is recommended to back the VM with hugepage sizes large enough so CMDQS stays at the HW maximum. Smaller backing pages reduce CMDQS accordingly.
--- hw/arm/tegra241-cmdqv.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
hw/arm/tegra241-cmdqv.c +20 -0
@@ -16,6 +16,8 @@#include "hw/arm/smmuv3-common.h" #include "hw/core/irq.h" #include "smmuv3-accel.h" +#include "smmuv3-internal.h" +#include "system/hostmem.h" #include "tegra241-cmdqv.h" #include "trace.h"
@@ -874,6 +876,8 @@ free_viommu:static void tegra241_cmdqv_init_regs (SMMUv3State *s,Tegra241CMDQV *cmdqv) {int i; +long pgsize; +uint32_t val; cmdqv->config =V_CONFIG_RESET ; cmdqv->param =FIELD_DP32 (0 ,PARAM ,CMDQV_VER ,CMDQV_VER );
@@ -905,6 +909,22 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) cmdqv->vcmdq_base[i] =0 ; cmdqv->vcmdq_cons_indx_base[i] =0 ; } + +/* +* CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the +* queue fits entirely within the smallest backend page size, ensuring +* the command queue is physically contiguous in host memory. +* +* IDR1.CMDQS = log2(max_qsz) - entry_shift +* +* where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4). +*/ + pgsize =qemu_minrampagesize (); +if (pgsize ==LONG_MAX ) { + pgsize =qemu_real_host_page_size (); + } + val =FIELD_EX32 (s->idr[1 ],IDR1 ,CMDQS ); + s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,CMDQS ,MIN (ctz64 (pgsize) -4 , val)); }static void tegra241_cmdqv_reset (SMMUv3State *s)
Signature · 1 line
2.43.0
Re: [PATCH v6 25/31] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size
Quoted 14 lines · Shameer Kolothum
From: Nicolin Chen <[email protected]>
CMDQV HW performs DMA accesses to guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.
Use
qemu_minrampagesize()to find the smallest memory-backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.
Quoted 8 lines
Note this is done at SMMUv3 init, before any guest queue GPA is known, so the cap is conservative. Maximum queue size is 8MiB; it is recommended to back the VM with hugepage sizes large enough so CMDQS stays at the HW maximum. Smaller backing pages reduce CMDQS accordingly.
Signed-off-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Eric
Quoted 49 lines
--- hw/arm/tegra241-cmdqv.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)hw/arm/tegra241-cmdqv.c +20
-0@@ -16,6 +16,8 @@#include "hw/arm/smmuv3-common.h" #include "hw/core/irq.h" #include "smmuv3-accel.h" +#include "smmuv3-internal.h" +#include "system/hostmem.h" #include "tegra241-cmdqv.h" #include "trace.h" @@ -874,6 +876,8 @@ free_viommu:static void tegra241_cmdqv_init_regs (SMMUv3State *s,Tegra241CMDQV *cmdqv) {int i; +long pgsize; +uint32_t val; cmdqv->config =V_CONFIG_RESET ; cmdqv->param =FIELD_DP32 (0 ,PARAM ,CMDQV_VER ,CMDQV_VER );@@ -905,6 +909,22 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) cmdqv->vcmdq_base[i] =0 ; cmdqv->vcmdq_cons_indx_base[i] =0 ; } + +/* +* CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the +* queue fits entirely within the smallest backend page size, ensuring +* the command queue is physically contiguous in host memory. +* +* IDR1.CMDQS = log2(max_qsz) - entry_shift +* +* where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4). +*/ + pgsize =qemu_minrampagesize (); +if (pgsize ==LONG_MAX ) { + pgsize =qemu_real_host_page_size (); + } + val =FIELD_EX32 (s->idr[1 ],IDR1 ,CMDQS ); + s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,CMDQS ,MIN (ctz64 (pgsize) -4 , val)); }static void tegra241_cmdqv_reset (SMMUv3State *s)
PATCH v6 26/31 hw/arm/smmuv3: Add per-device identifier property
In reply to Shameer KolothumAdd an "identifier" property to the SMMUv3 device and use it when building the ACPI IORT SMMUv3 node Identifier field.
This avoids relying on device enumeration order and provides a stable per-device identifier. A subsequent patch will use the same identifier when generating the DSDT description for Tegra241 CMDQV, ensuring that the IORT and DSDT entries refer to the same SMMUv3 instance.
The identifier is assigned at pre-plug time, accounting for the ITS Group node that build_iort() places before SMMUv3 nodes in the IORT table, so that identifiers are globally unique across all IORT nodes.
No functional change: IORT blob content for bios-tables qtest is identical to before.
--- include/hw/arm/smmuv3.h | 1 + hw/arm/smmuv3.c | 2 ++ hw/arm/virt-acpi-build.c | 5 ++++- hw/arm/virt.c | 12 ++++++++++++ 4 files changed, 19 insertions(+), 1 deletion(-)
include/hw/arm/smmuv3.h +1 -0
@@ -65,6 +65,7 @@ struct SMMUv3State {qemu_irq irq[4 ];QemuMutex mutex;char *stage; +uint8_t identifier;/* SMMU has HW accelerator support for nested S1 + s2 */ bool accel;
hw/arm/smmuv3.c +2 -0
@@ -2126,6 +2126,8 @@ static const Property smmuv3_properties[] = {* Defaults to stage 1 */ DEFINE_PROP_STRING ("stage" , SMMUv3State, stage), +/* Identifier used for ACPI IORT SMMUv3 (and DSDT for CMDQV) generation */ +DEFINE_PROP_UINT8 ("identifier" , SMMUv3State, identifier,0 ),DEFINE_PROP_BOOL ("accel" , SMMUv3State, accel, false),/* GPA of MSI doorbell, for SMMUv3 accel use. */ DEFINE_PROP_UINT64("msi-gpa", SMMUv3State, msi_gpa, 0),
hw/arm/virt-acpi-build.c +4 -1
@@ -349,6 +349,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b) typedef struct AcpiIortSMMUv3Dev { int irq; hwaddr base; + uint8_t id; GArray *rc_smmu_idmaps; /* Offset of the SMMUv3 IORT Node relative to the start of the IORT */ size_t offset;
@@ -411,6 +412,7 @@ static int populate_smmuv3_dev(VirtMachineState *vms, GArray *sdev_blob) &error_abort)); sdev.accel =object_property_get_bool (obj,"accel" , &error_abort); sdev.ats =smmuv3_ats_enabled (ARM_SMMUV3 (obj)); + sdev.id =object_property_get_uint (obj,"identifier" , &error_abort); pbus =PLATFORM_BUS_DEVICE (vms->platform_bus_dev); sbdev =SYS_BUS_DEVICE (obj); sdev.base =platform_bus_get_mmio_addr (pbus, sbdev,0 );
@@ -637,7 +639,8 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) (ID_MAPPING_ENTRY_SIZE * smmu_mapping_count);build_append_int_noprefix (table_data, node_size,2 );/* Length */ build_append_int_noprefix (table_data,4 ,1 );/* Revision */ -build_append_int_noprefix (table_data, id++,4 );/* Identifier */ +build_append_int_noprefix (table_data, sdev->id,4 );/* Identifier */ + id++;/* advance shared counter for RC/RMR node uniqueness */ /* Number of ID mappings */ build_append_int_noprefix (table_data, smmu_mapping_count,4 );/* Reference to ID Array */
hw/arm/virt.c +12 -0
@@ -254,6 +254,9 @@ static MemMapEntry extended_memmap[] = {/* Any CXL Fixed memory windows come here */ }; +/* Counts SMMUv3 devices plugged; used to assign stable IORT identifiers */ +static uint8_t smmuv3_dev_id; +static const int a15irqmap[] = { [VIRT_UART0 ] =1 , [VIRT_RTC ] =2 ,
@@ -3830,6 +3833,15 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, OBJECT(vms->sysmem), NULL);object_property_set_link (OBJECT (dev),"secure-memory" ,OBJECT (vms->secure_sysmem),NULL ); +/* +* In build_iort(), the ITS node(id=0) precedes SMMUv3 nodes +* when present. Account for it so this SMMUv3's identifier +* is globally unique across all IORT nodes. +*/ +uint8_t its_offset = (vms->msi_controller ==VIRT_MSI_CTRL_ITS ) + ?1 :0 ; +object_property_set_uint (OBJECT (dev),"identifier" , + its_offset + smmuv3_dev_id++,NULL ); }if (object_property_get_bool (OBJECT (dev),"accel" , &error_abort)) {hwaddr db_start =0 ;
Signature · 1 line
2.43.0
PATCH v6 27/31 hw/arm/smmuv3-accel: Introduce helper to query CMDQV type
In reply to Shameer KolothumIntroduce a SMMUv3AccelCmdqvType enum and a helper to query the CMDQV implementation type associated with an accelerated SMMUv3 instance.
A subsequent patch will use this helper when generating the Tegra241 CMDQV DSDT.
--- hw/arm/smmuv3-accel.h | 10 ++++++++++ hw/arm/smmuv3-accel-stubs.c | 5 +++++ hw/arm/smmuv3-accel.c | 12 ++++++++++++ hw/arm/tegra241-cmdqv.c | 6 ++++++ 4 files changed, 33 insertions(+)
hw/arm/smmuv3-accel.h +10 -0
@@ -16,6 +16,11 @@#include <linux/iommufd.h> #endif +typedef enum SMMUv3AccelCmdqvType { +SMMUV3_CMDQV_NONE =0 , +SMMUV3_CMDQV_TEGRA241 , +}SMMUv3AccelCmdqvType ; +/* * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to * support multiple VCMDQs with virtualization capabilities.
@@ -42,6 +47,10 @@ typedef struct SMMUv3AccelCmdqvOps {* If NULL, the viommu_id is freed directly via iommufd_backend_free_id(). */ void (*free_viommu)(SMMUv3State *s); +/** +* @get_type: Optional callback. Return the CMDQV implementation type. +*/ +SMMUv3AccelCmdqvType (*get_type)(void);/** * @reset: Optional callback. Reset CMDQV state. */
@@ -90,5 +99,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);bool smmuv3_accel_event_read_validate (IOMMUFDVeventq *veventq,uint32_t type,void *buf,size_t size,Error **errp);void smmuv3_accel_reset (SMMUv3State *s); +SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type (Object *obj);#endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +5 -0
@@ -57,3 +57,8 @@ bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,void smmuv3_accel_reset (SMMUv3State *s) { } + +SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type (Object *obj) +{ +return SMMUV3_CMDQV_NONE ; +}
hw/arm/smmuv3-accel.c +12 -0
@@ -1060,6 +1060,18 @@ static void smmuv3_accel_as_init(SMMUv3State *s)address_space_init (shared_as_sysmem, &root,"smmuv3-accel-as-sysmem" ); } +SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type (Object *obj) +{ +SMMUv3State *s =ARM_SMMUV3 (obj); +SMMUv3AccelState *accel = s->s_accel; + +if (!accel || !accel->cmdqv_ops || !accel->cmdqv_ops->get_type) { +return SMMUV3_CMDQV_NONE ; + } + +return accel->cmdqv_ops->get_type (); +} +static void smmuv3_accel_machine_done (Notifier *notifier,void *data) {SMMUv3State *s =container_of (notifier, SMMUv3State, machine_done);
hw/arm/tegra241-cmdqv.c +6 -0
@@ -974,6 +974,11 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)return true; } +static SMMUv3AccelCmdqvType tegra241_cmdqv_get_type (void ) +{ +return SMMUV3_CMDQV_TEGRA241 ; +} +static bool tegra241_cmdqv_probe (SMMUv3State *s,HostIOMMUDeviceIOMMUFD *idev,Error **errp) {
@@ -1014,6 +1019,7 @@ static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = { .init = tegra241_cmdqv_init, .alloc_viommu = tegra241_cmdqv_alloc_viommu, .free_viommu = tegra241_cmdqv_free_viommu, + .get_type = tegra241_cmdqv_get_type, .reset = tegra241_cmdqv_reset, };
Signature · 1 line
2.43.0
PATCH v6 28/31 hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT
In reply to Shameer KolothumFrom: Nicolin Chen <[email protected]>
Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv.
For each accelerated SMMUv3 instance, add a Tegra241 CMDQV device object under the DSDT \_SB namespace, with HID "NVDA200C" and a UID that matches the Identifier of the corresponding SMMUv3 IORT node, so the guest OS can associate the DSDT device with the right SMMU. The _CRS covers the CMDQV MMIO aperture plus its interrupt, and _CCA declares I/O cache coherency.
See ACPI Specification 6.5, Section 6 (Device Configuration) for _HID/_UID/_CCA/_CRS.
Generated DSDT entry for a CMDQV instance paired with SMMUv3 Identifier=1:
...
Device (CV01)
{
Name (_HID, "NVDA200C") // _HID: Hardware ID
Name (_UID, One) // _UID: Unique ID
Name (_CCA, One) // _CCA: Cache Coherency Attribute
Name (_CRS, ResourceTemplate () // _CRS: Current Resource Settings
{
QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite,
0x0000000000000000, // Granularity
0x000000000C080000, // Range Minimum
0x000000000C0CFFFF, // Range Maximum
0x0000000000000000, // Translation Offset
0x0000000000050000, // Length
,, , AddressRangeMemory, TypeStatic)
Interrupt (ResourceConsumer, Edge, ActiveHigh, Exclusive, ,, )
{
0x00000094,
}
}) }
...
Generated IORT SMMUv3 node (Identifier = 1):
...
[048h 0072 001h] Type : 04
[049h 0073 002h] Length : 0058
[04Bh 0075 001h] Revision : 04
[04Ch 0076 004h] Identifier : 00000001
[050h 0080 004h] Mapping Count : 00000001
[054h 0084 004h] Mapping Offset : 00000044
...
--- hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++ hw/arm/trace-events | 1 + 2 files changed, 53 insertions(+)
hw/arm/virt-acpi-build.c +52 -0
@@ -65,6 +65,9 @@#include "target/arm/cpu.h" #include "target/arm/multiprocessing.h" +#include "smmuv3-accel.h" +#include "tegra241-cmdqv.h" +#define ARM_SPI_BASE 32#define ACPI_BUILD_TABLE_SIZE 0x20000
@@ -1121,6 +1124,51 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,build_fadt (table_data, linker, &fadt, vms->oem_id, vms->oem_table_id); } +static void acpi_dsdt_add_tegra241_cmdqv (Aml *scope,VirtMachineState *vms) +{ +for (int i =0 ; i < vms->smmuv3_devices->len; i++) { +Object *obj =OBJECT (g_ptr_array_index (vms->smmuv3_devices, i)); +PlatformBusDevice *pbus; +Aml *dev, *crs, *addr; +SysBusDevice *sbdev; +hwaddr base; +uint32_t id; +int irq; + +if (smmuv3_accel_cmdqv_type (obj) !=SMMUV3_CMDQV_TEGRA241 ) { +continue ; + } + id =object_property_get_uint (obj,"identifier" , &error_abort); + pbus =PLATFORM_BUS_DEVICE (vms->platform_bus_dev); + sbdev =SYS_BUS_DEVICE (obj); + base =platform_bus_get_mmio_addr (pbus, sbdev,1 ); + base += vms->memmap[VIRT_PLATFORM_BUS ].base; + irq =platform_bus_get_irqn (pbus, sbdev,NUM_SMMU_IRQS ); + irq += vms->irqmap[VIRT_PLATFORM_BUS ]; + irq +=ARM_SPI_BASE ; + + dev =aml_device ("CV%.02u" , id); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("NVDA200C" ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (id))); +aml_append (dev,aml_name_decl ("_CCA" ,aml_int (1 ))); + + crs =aml_resource_template (); + addr =aml_qword_memory (AML_POS_DECODE ,AML_MIN_FIXED ,AML_MAX_FIXED , +AML_CACHEABLE ,AML_READ_WRITE ,0x0 , base, + base +TEGRA241_CMDQV_IO_LEN -0x1 ,0x0 , +TEGRA241_CMDQV_IO_LEN ); +aml_append (crs, addr); +aml_append (crs,aml_interrupt (AML_CONSUMER ,AML_EDGE , +AML_ACTIVE_HIGH ,AML_EXCLUSIVE , + (uint32_t *)&irq,1 )); +aml_append (dev,aml_name_decl ("_CRS" , crs)); + +aml_append (scope, dev); + +trace_virt_acpi_dsdt_tegra241_cmdqv (id, base, irq); + } +} +/* DSDT */ static void build_dsdt (GArray *table_data,BIOSLinker *linker,VirtMachineState *vms)
@@ -1185,6 +1233,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)acpi_dsdt_add_tpm (scope, vms);#endif +if (!vms->legacy_smmuv3_present) { +acpi_dsdt_add_tegra241_cmdqv (scope, vms); + } +aml_append (dsdt, scope); pci0_scope =aml_scope ("\\_SB.PCI0" );
hw/arm/trace-events +1 -0
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s" # virt-acpi-build.c virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out." +virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d" # smmu-common.c smmu_add_mr(const char *name) "%s"
Signature · 1 line
2.43.0
PATCH v6 29/31 hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active
In reply to Shameer KolothumWhen CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association, and the guest's boot-time CMDQV configuration (VINTFs, VCMDQs) is built on top of that association.
Hot-unplugging that device would release the viommu and tear down all CMDQV state. Hot-plugging another device behind a different host SMMUv3+CMDQV would then re-bind the same vSMMUv3 to new host hardware, while the guest keeps using its boot-time configuration and ends up issuing commands to the wrong host. Block hot-unplug of the establishing device to avoid this; retaining the binding across unplug is non-trivial and not required by any current use case.
Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.
--- hw/arm/smmuv3-accel.h | 1 + hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
hw/arm/smmuv3-accel.h +1 -0
@@ -84,6 +84,7 @@ typedef struct SMMUv3AccelDevice {IOMMUFDVdev *vdev;QLIST_ENTRY (SMMUv3AccelDevice) next;SMMUv3AccelState *s_accel; +Error *unplug_blocker;/* set when CMDQV is active to block hot-unplug */ } SMMUv3AccelDevice;bool smmuv3_accel_init (SMMUv3State *s,Error **errp);
hw/arm/smmuv3-accel.c +18 -0
@@ -759,6 +759,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,return false; } +/* +* CMDQV is active: block hot-unplug of the device that established the +* viommu association. Removing it would cause the vIOMMU to host SMMUv3 +* association be changed via device hot-plug. +*/ +if (s->s_accel->cmdqv_ops) { +PCIDevice *pdev =pci_find_device (bus,pci_bus_num (bus), devfn); +error_setg (&accel_dev->unplug_blocker, +"CMDQV is active: removing the device that established the " +"viommu association would break the guest CMDQV" ); +qdev_add_unplug_blocker (DEVICE (pdev), accel_dev->unplug_blocker); + } done: accel_dev->hiodi = hiodi; accel_dev->s_accel = s->s_accel;
@@ -1082,6 +1094,12 @@ static void smmuv3_accel_machine_done(Notifier *notifier, void *data) "at least one cold-plugged VFIO device");exit (1 ); } + +if (s->cmdqv ==ON_OFF_AUTO_ON && !accel->cmdqv) { +error_report ("arm-smmuv3 cmdqv=on requires at least one cold-plugged " +"VFIO device" ); +exit (1 ); + } }bool smmuv3_accel_init (SMMUv3State *s,Error **errp)
Signature · 1 line
2.43.0
Re: [PATCH v6 29/31] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active
Quoted 12 lines · Shameer Kolothum
When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association, and the guest's boot-time CMDQV configuration (VINTFs, VCMDQs) is built on top of that association.
Hot-unplugging that device would release the viommu and tear down all CMDQV state. Hot-plugging another device behind a different host SMMUv3+CMDQV would then re-bind the same vSMMUv3 to new host hardware, while the guest keeps using its boot-time configuration and ends up issuing commands to the wrong host. Block hot-unplug of the establishing device to avoid this; retaining the binding across unplug is non-trivial and not required by any current use case.
Quoted 5 lines
Also abort at
machine_doneif cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.Reviewed-by Nicolin Chen <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Eric
Quoted 53 lines
--- hw/arm/smmuv3-accel.h | 1 + hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)hw/arm/smmuv3-accel.h +1
-0@@ -84,6 +84,7 @@ typedef struct SMMUv3AccelDevice {IOMMUFDVdev *vdev;QLIST_ENTRY (SMMUv3AccelDevice) next;SMMUv3AccelState *s_accel; +Error *unplug_blocker;/* set when CMDQV is active to block hot-unplug */ } SMMUv3AccelDevice;bool smmuv3_accel_init (SMMUv3State *s,Error **errp);hw/arm/smmuv3-accel.c +18
-0@@ -759,6 +759,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,return false; } +/* +* CMDQV is active: block hot-unplug of the device that established the +* viommu association. Removing it would cause the vIOMMU to host SMMUv3 +* association be changed via device hot-plug. +*/ +if (s->s_accel->cmdqv_ops) { +PCIDevice *pdev =pci_find_device (bus,pci_bus_num (bus), devfn); +error_setg (&accel_dev->unplug_blocker, +"CMDQV is active: removing the device that established the " +"viommu association would break the guest CMDQV" ); +qdev_add_unplug_blocker (DEVICE (pdev), accel_dev->unplug_blocker); + } done: accel_dev->hiodi = hiodi; accel_dev->s_accel = s->s_accel;@@ -1082,6 +1094,12 @@ static void smmuv3_accel_machine_done(Notifier *notifier, void *data) "at least one cold-plugged VFIO device");exit (1 ); } + +if (s->cmdqv ==ON_OFF_AUTO_ON && !accel->cmdqv) { +error_report ("arm-smmuv3 cmdqv=on requires at least one cold-plugged " +"VFIO device" ); +exit (1 ); + } }bool smmuv3_accel_init (SMMUv3State *s,Error **errp)
PATCH v6 30/31 hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle
In reply to Shameer KolothumAdd an overview describing the Tegra241 CMDQV passthrough model, MMIO layout, guest-driven lifecycle, and per-VM isolation.
--- hw/arm/tegra241-cmdqv.c | 93 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+)
hw/arm/tegra241-cmdqv.c +93 -0
@@ -7,6 +7,99 @@* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Tegra241 CMDQV - overview +* ========================= +* +* NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V) +* block. It lets a guest issue SMMU invalidation commands directly to +* dedicated hardware queues (vCMDQs) without trapping into the hypervisor on +* the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the +* host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd. +* QEMU emulates the CMDQV MMIO region and drives the host kernel calls +* (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing +* happens on real hardware. +* +* A vCMDQ becomes functional only once allocated to a VINTF; until then, +* register accesses only touch the QEMU-side cache and no command +* processing happens. After allocation, command processing runs on the +* host hardware, and guest accesses to the live control/status registers +* bypass QEMU and reach the host directly. +* +* MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) +* ----------------------------------------------------- +* The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF +* apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots +* under different addressing. +* +* 0x00000 CMDQV Config page: QEMU-trapped. +* 0x10000 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed +* to either the mmap'd VINTF Page 0 (if the vCMDQ has been +* allocated to a VINTF) or a per-vCMDQ register cache (otherwise). +* 0x20000 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped. +* 0x30000 VINTF Page 0 (per-VINTF control/status): mmap'd from the host +* via iommufd and installed into guest MMIO as a RAM-device +* subregion after the first HW_QUEUE_ALLOC; subsequent accesses +* bypass QEMU. +* 0x40000 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. Although this is +* a HW alias of the direct Page 1, the kernel only exposes mmap +* for VINTF Page 0; VINTF Page 1 is not mmap'd and stays trapped. +* +* The direct vCMDQ Page 0 stays trapped rather than aliased to the VINTF +* Page 0 mmap. Direct aperture read/write is not an expected common usecase, +* but trapping keeps accesses well-defined for an unallocated vCMDQ. +* +* Lifecycle (driven by guest events) +* ---------------------------------- +* 1. First vfio-pci device attach (.set_iommu_device) triggers: +* - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support. +* - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, +* configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, +* forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0. +* +* 2. Guest writes VINTF_CONFIG.ENABLE = 1: +* QEMU mmap()s the offset from step 1 into its address space and reports +* STATUS.ENABLE_OK = 1. The host VINTF was already enabled by +* IOMMU_VIOMMU_ALLOC; QEMU only acks back. +* +* 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN, +* VINTF.ENABLE, in any order; each precondition write retries the +* allocation): +* IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through +* stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF. +* +* 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is +* installed into guest MMIO as a RAM-device subregion. Guest VINTF Page 0 +* accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter +* go straight to host hardware, bypassing QEMU. +* +* 5. Guest SMMU driver programs a Stream Table Entry for a passthrough +* device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE in this +* VM's VINTF so the device's guest vSID translates to its host pSID. +* Commands referencing unmapped SIDs are rejected by HW. +* +* Per-VM isolation +* ---------------- +* - Each VM has its own iommufd FD; all iommufd objects (VINTF, vdevices, +* hw_queues, mmap regions) belong to that FD. Cross-FD lookups fail, so +* one VM cannot reach another VM's IDs. +* - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV +* hardware substitutes / checks VMID on every command the guest issues. +* - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the +* guest to a safe subset of commands. +* - IOMMU_VDEVICE_ALLOC populates SID_MATCH/SID_REPLACE so invalidations +* only reach the host StreamIDs assigned to this VM (see step 5). +* - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest +* cannot reach a vCMDQ that belongs to another VM. +* +* Limits exposed to the guest +* --------------------------- +* One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. Maximum vCMDQ +* size is 8MiB. The queue must be physically contiguous (the HW reads it +* via host PA), so QEMU caps it to the host memory-backend page size. Use +* hugepage backing large enough to keep CMDQS at the HW maximum. +*/ +#include "qemu/osdep.h" #include "qemu/log.h" #include "qemu/error-report.h"
Signature · 1 line
2.43.0
Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle
Quoted 9 lines · Shameer Kolothum
+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ * -tegra241_cmdqv_probe():IOMMU_GET_HW_INFOconfirms host CMDQV support.
+ * -IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ * configures the VM's VMID (from its stage-2 HWPT) inVINTF_CONFIG,
+ * forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
Quoted 3 lines
+ * QEMU mmap()s the offset from step 1 into its address space and reports + * STATUS.ENABLE_OK = 1. The host VINTF was already enabled by + * IOMMU_VIOMMU_ALLOC; QEMU only acks back.
I wonder this mmap should be done after the CMDQV_CONFIG.CMDQV_EN=1 rather than VINTF_CONFIG.ENABLE=1.
Spec says that:
"Program the CMDQV_CMDQ_ALLOC_MAP_X register(s) to map the Virtual
CMDQ(s) to the logical CMDQs on Virtual Interface following the CMDQ allocation rules. This again is an optional step and the CMDQ allocation can be done after the Virtual Interface is initialized."
This means the LVCMDQ mapping can happen before VINTF_CONFIG.ENABLE=1.
And the mmap info is returned with IOMMU_VIOMMU_ALLOC. So, this is doable?
FWIW, kernel finalizes a LVCMDQ mapping when handling HW_QUEUE_ALLOC.
Quoted 10 lines
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC,
CMDQV_EN,+ * VINTF.ENABLE, in any order; each precondition write retries the + * allocation): + * IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through + * stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF. + * + * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is + * installed into guest MMIO as a RAM-device subregion. Guest VINTF Page 0 + * accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter + * go straight to host hardware, bypassing QEMU.
Quoted 3 lines
+ *
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ * device:IOMMU_VDEVICE_ALLOCprograms SID_MATCH/SID_REPLACE in this
Ideally, the VDEVICE_ALLOC should happen right after the device gets plugged *and* its vSID is available; similarly the vdevice should be only freed when the device gets unplugged. So, its lifecycle should stay with the plugged state.
We do VDEVICE_ALLOC when guest programming STE is because that's the only place we found in the existing functions. We could (or arguably maybe we should) find a better place to allocate VDEVICE. E.g., vfio code can call qemu_add_vm_change_state_handler_prio() to register a callback function, where vSID (vBDF) should be available once the VM start to be "running".
I think your narrative fits the current design. But we might need to be more accurate here, since this is about "lifecycle".
Quoted 1 line
+ * VM's VINTF so the device's guest vSID translates to its host pSID.
Nit: so that the HW translates the guest vSID into host pSID.
Quoted 6 lines
+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. Maximum vCMDQ
+ * size is 8MiB. The queue must be physically contiguous (the HW reads it
+ * via host PA), so QEMU caps it to the host memory-backend page size. Use
+ * hugepage backing large enough to keep CMDQS at the HW maximum.
The "physically contiguous" limit is handled by QEMU, not exposed to the guest at all; guest is unaware of this.
Nicolin
RE: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle
Quoted 42 lines
-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 02 June 2026 00:47
To: Shameer Kolothum Thodi <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycleQuoted 4 lines · Shameer Kolothum
+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ * -tegra241_cmdqv_probe():IOMMU_GET_HW_INFOconfirms hostCMDQV support.
Quoted 3 lines
+ * -
IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ * configures the VM's VMID (from its stage-2 HWPT) inVINTF_CONFIG,
+ * forces HYP_OWN=0, and returns the mmap offset/length for VINTFPage 0.
Quoted 3 lines
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ * QEMU mmap()s the offset from step 1 into its address space andreports
Quoted 2 lines
+ * STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *IOMMU_VIOMMU_ALLOC; QEMU only acks back.I wonder this mmap should be done after the
CMDQV_CONFIG.CMDQV_EN=1
rather than VINTF_CONFIG.ENABLE=1.Spec says that:
"Program theCMDQV_CMDQ_ALLOC_MAP_Xregister(s) to map the Virtual
CMDQ(s) to the logical CMDQs on Virtual Interface following the CMDQ allocation rules. This again is an optional step and the CMDQ allocation can be done after the Virtual Interface is initialized."
Quoted 5 lines
This means the LVCMDQ mapping can happen before
VINTF_CONFIG.ENABLE=1.And the mmap info is returned with
IOMMU_VIOMMU_ALLOC. So, this is doable?
I think it is. We could in theory as well do it just after IOMMU_VIOMMU_ALLOC. What you think?
Quoted 36 lines
FWIW, kernel finalizes a LVCMDQ mapping when handling
HW_QUEUE_ALLOC.Quoted 1 line
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC,
CMDQV_EN,Quoted 3 lines
+ * VINTF.ENABLE, in any order; each precondition write retries the + * allocation): + * IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translatedthrough
Quoted 3 lines
+ * stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
+ *
+ * 4. After the first successfulHW_QUEUE_ALLOC, the mmap'd VINTF Page0 is
Quoted 1 line
+ * installed into guest MMIO as a RAM-device subregion. Guest VINTF
Page 0
Quoted 1 line
+ * accesses (
CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN)thereafter
Quoted 4 lines
+ * go straight to host hardware, bypassing QEMU.
+ *
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ * device:IOMMU_VDEVICE_ALLOCprograms SID_MATCH/SID_REPLACEin this
Ideally, the
VDEVICE_ALLOCshould happen right after the device gets plugged *and* its vSID is available; similarly the vdevice should be only freed when the device gets unplugged. So, its lifecycle should stay with the plugged state.We do
VDEVICE_ALLOCwhen guest programming STE is because that's the only place we found in the existing functions. We could (or arguably maybe we should) find a better place to allocate VDEVICE. E.g., vfio code can callqemu_add_vm_change_state_handler_prio()to register a callback function, where vSID (vBDF) should be available once the VM start to be "running".
Quoted 2 lines
I think your narrative fits the current design. But we might need to be more accurate here, since this is about "lifecycle".
Yes, the above is based on the current design. I will add a caveat to cover that.
I remember discussions about use cases(CCA realm ?) requiring an early mandatory VDEVICE_ALLOC before guest boot. Regarding the suggestion to use qemu_add_vm_change_state_handler_prio(), it might help, but I'm not sure it covers all the cases like, hotplug, guest PCI re-enumeration during boot, etc. (Having _DSM #5 might help here, but per the PCI Firmware Spec it only "indicates to an operating system that it should preserve assignments of PCI bus numbers", so, guests are not strictly required to honour it.) Anyway, I'll take another look as a follow up.
Quoted 7 lines
Quoted 1 line
+ * VM's VINTF so the device's guest vSID translates to its host pSID.
Nit: so that the HW translates the guest vSID into host pSID.
Quoted 3 lines
+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF.
Quoted 8 lines
Maximum vCMDQ
Quoted 2 lines
+ * size is 8MiB. The queue must be physically contiguous (the HW reads it
+ * via host PA), so QEMU caps it to the host memory-backend page size.Use
Quoted 1 line
+ * hugepage backing large enough to keep CMDQS at the HW maximum.
The "physically contiguous" limit is handled by QEMU, not exposed to the guest at all; guest is unaware of this.
Ok. The intend was the queue size exposed may be limited because of the "physically contiguous" requirement. I will make it clear.
Thanks,
Shameer
Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle
Quoted 17 lines · Shameer Kolothum Thodi
Quoted 17 lines
From: Nicolin Chen <[email protected]>
Quoted 2 lines
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ * QEMU mmap()s the offset from step 1 into its address space andreports
Quoted 2 lines
+ * STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *IOMMU_VIOMMU_ALLOC; QEMU only acks back.I wonder this mmap should be done after the
CMDQV_CONFIG.CMDQV_EN=1
rather than VINTF_CONFIG.ENABLE=1.Spec says that:
"Program theCMDQV_CMDQ_ALLOC_MAP_Xregister(s) to map the Virtual
CMDQ(s) to the logical CMDQs on Virtual Interface following the CMDQ allocation rules. This again is an optional step and the CMDQ allocation can be done after the Virtual Interface is initialized."
Quoted 8 lines
Quoted 5 lines
This means the LVCMDQ mapping can happen before
VINTF_CONFIG.ENABLE=1.And the mmap info is returned with
IOMMU_VIOMMU_ALLOC. So, this is doable?I think it is. We could in theory as well do it just after
IOMMU_VIOMMU_ALLOC. What you think?
Yea. That sounds right to me.
Quoted 22 lines
Quoted 16 lines
Quoted 2 lines
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ * device:IOMMU_VDEVICE_ALLOCprograms SID_MATCH/SID_REPLACEin this
Ideally, the
VDEVICE_ALLOCshould happen right after the device gets plugged *and* its vSID is available; similarly the vdevice should be only freed when the device gets unplugged. So, its lifecycle should stay with the plugged state.We do
VDEVICE_ALLOCwhen guest programming STE is because that's the only place we found in the existing functions. We could (or arguably maybe we should) find a better place to allocate VDEVICE. E.g., vfio code can callqemu_add_vm_change_state_handler_prio()to register a callback function, where vSID (vBDF) should be available once the VM start to be "running".Quoted 2 lines
I think your narrative fits the current design. But we might need to be more accurate here, since this is about "lifecycle".
Yes, the above is based on the current design. I will add a caveat to cover that.
Quoted 9 lines
I remember discussions about use cases(CCA realm ?) requiring an early mandatory
VDEVICE_ALLOCbefore guest boot. Regarding the suggestion to useqemu_add_vm_change_state_handler_prio(), it might help, but I'm not sure it covers all the cases like, hotplug, guest PCI re-enumeration during boot, etc. (Having_DSM#5 might help here, but per the PCI Firmware Spec it only "indicates to an operating system that it should preserve assignments of PCI bus numbers", so, guests are not strictly required to honour it.) Anyway, I'll take another look as a follow up.
Yea. I am not giving a suggestion that would work perfectly here. I think we are good with what we implemented for now. REALM stuff will need to take another look.
Nicolin
Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle
In reply to Shameer KolothumHi Shammer,
Quoted 2 lines · Shameer Kolothum
Add an overview describing the Tegra241 CMDQV passthrough model, MMIO layout, guest-driven lifecycle, and per-VM isolation.
some suggestions inline. Thank you very much for the documentation efforts! That's really helpful.
Quoted 12 lines
Signed-off-by Shameer Kolothum <[email protected]>--- hw/arm/tegra241-cmdqv.c | 93 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+)hw/arm/tegra241-cmdqv.c +0
-0@@ -7,6 +7,99 @@* SPDX-License-Identifier: GPL-2.0-or-later
hw/arm/tegra241-cmdqv.c line 17
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +8
-0*/ +/* +* Tegra241 CMDQV - overview +* ========================= +* +* NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V) +* block. It lets a guest issue SMMU invalidation commands directly to +* dedicated hardware queues (vCMDQs) without trapping into the hypervisor on +* the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the
s/are grouped/are exclusively allocated to VINTFS?
hw/arm/tegra241-cmdqv.c line 25
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+* host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd. +* QEMU emulates the CMDQV MMIO region and drives the host kernel calls +* (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing +* happens on real hardware. +* +* A vCMDQ becomes functional only once allocated to a VINTF; until then, +* register accesses only touch the QEMU-side cache and no command +* processing happens. After allocation, command processing runs on the
state is migrated from cache to HW and command processing ...
hw/arm/tegra241-cmdqv.c line 33
Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8
-0+* host hardware, and guest accesses to the live control/status registers +* bypass QEMU and reach the host directly. +* +* MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) +* ----------------------------------------------------- +* The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF +* apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots +* under different addressing.
I would put above sentence after the layout description.
Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6
-0+* +* 0x00000 CMDQV Config page: QEMU-trapped. +* 0x10000 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed +* to either the mmap'd VINTF Page 0 (if the vCMDQ has been +* allocated to a VINTF) or a per-vCMDQ register cache (otherwise). +* 0x20000 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped.
hw/arm/tegra241-cmdqv.c line 49
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+* 0x30000 VINTF Page 0 (per-VINTF control/status): mmap'd from the host +* via iommufd and installed into guest MMIO as a RAM-device +* subregion after the first HW_QUEUE_ALLOC; subsequent accesses +* bypass QEMU. +* 0x40000 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. Although this is +* a HW alias of the direct Page 1, the kernel only exposes mmap +* for VINTF Page 0; VINTF Page 1 is not mmap'd and stays trapped. +* +* The direct vCMDQ Page 0 stays trapped rather than aliased to the VINTF +* Page 0 mmap. Direct aperture read/write is not an expected common usecase,
You can justify it as you did in the individual patch (to keep management of unmapped vcmds in direct page compliant with the spec)
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4
-0+* but trapping keeps accesses well-defined for an unallocated vCMDQ. +* +* Lifecycle (driven by guest events) +* ----------------------------------
hw/arm/tegra241-cmdqv.c line 63
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10
-0+* 1. First vfio-pci device attach (.set_iommu_device) triggers: +* - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support. +* - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, +* configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, +* forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0. +* +* 2. Guest writes VINTF_CONFIG.ENABLE = 1: +* QEMU mmap()s the offset from step 1 into its address space and reports +* STATUS.ENABLE_OK = 1. The host VINTF was already enabled by +* IOMMU_VIOMMU_ALLOC; QEMU only acks back.
the host VINTF ... : I would add this in 1. I would remove "QEMU only acks back."
hw/arm/tegra241-cmdqv.c line 66
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+* +* 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN, +* VINTF.ENABLE, in any order; each precondition write retries the
the HW queue allocation:
hw/arm/tegra241-cmdqv.c line 69
Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3
-0+* allocation): +* IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through +* stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
the guest is granted a new host vcmdq in its vintf?
Quoted 26 lines
hw/arm/tegra241-cmdqv.c +26
-0+* +* 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is +* installed into guest MMIO as a RAM-device subregion. Guest VINTF Page 0 +* accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter +* go straight to host hardware, bypassing QEMU. +* +* 5. Guest SMMU driver programs a Stream Table Entry for a passthrough +* device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE in this +* VM's VINTF so the device's guest vSID translates to its host pSID. +* Commands referencing unmapped SIDs are rejected by HW. +* +* Per-VM isolation +* ---------------- +* - Each VM has its own iommufd FD; all iommufd objects (VINTF, vdevices, +* hw_queues, mmap regions) belong to that FD. Cross-FD lookups fail, so +* one VM cannot reach another VM's IDs. +* - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV +* hardware substitutes / checks VMID on every command the guest issues. +* - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the +* guest to a safe subset of commands. +* - IOMMU_VDEVICE_ALLOC populates SID_MATCH/SID_REPLACE so invalidations +* only reach the host StreamIDs assigned to this VM (see step 5). +* - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest +* cannot reach a vCMDQ that belongs to another VM. +* +* Limits exposed to the guest
hw/arm/tegra241-cmdqv.c line 105
Quoted 10 lines
hw/arm/tegra241-cmdqv.c +7
-0+* --------------------------- +* One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. Maximum vCMDQ +* size is 8MiB. The queue must be physically contiguous (the HW reads it +* via host PA), so QEMU caps it to the host memory-backend page size. Use +* hugepage backing large enough to keep CMDQS at the HW maximum. +*/ +#include "qemu/osdep.h" #include "qemu/log.h" #include "qemu/error-report.h"
Thanks
Eric
PATCH v6 31/31 hw/arm/smmuv3: Add cmdqv property for SMMUv3 device
In reply to Shameer KolothumIntroduce a "cmdqv" property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.
--- hw/arm/smmuv3.c | 8 ++++++++ qemu-options.hx | 8 ++++++++ 2 files changed, 16 insertions(+)
hw/arm/smmuv3.c +8 -0
@@ -1994,6 +1994,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp) "bits if accel=on");return false; } +if (s->cmdqv ==ON_OFF_AUTO_ON ) { +error_setg (errp,"cmdqv can only be enabled if accel=on" ); +return false; + }return true; }
@@ -2144,6 +2148,7 @@ static const Property smmuv3_properties[] = {DEFINE_PROP_OAS_MODE ("oas" , SMMUv3State, oas,OAS_MODE_AUTO ),DEFINE_PROP_SSIDSIZE_MODE ("ssidsize" , SMMUv3State, ssidsize,SSID_SIZE_MODE_AUTO ), + DEFINE_PROP_ON_OFF_AUTO("cmdqv", SMMUv3State, cmdqv, ON_OFF_AUTO_AUTO), };static void smmuv3_instance_init (Object *obj)
@@ -2194,6 +2199,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"than 0 is required to enable PASID support." "Please ensure the value does not exceed the maximum " "SubtreamID size supported by the host platform."); +object_class_property_set_description (klass,"cmdqv" , +"Enable/disable CMDQV support (for accel=on). " +"Valid values are on, off, and auto. Defaults to auto." ); } static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
qemu-options.hx +8 -0
@@ -1329,6 +1329,14 @@ SRST - With accel=off, auto is resolved to 0. + ``cmdqv=on|off|auto`` (default: auto) + Enable hardware Command Queue Virtualization (CMDQV) for the + SMMUv3 command queue. Currently only the NVIDIA Tegra241 CMDQV + implementation is supported. + + - With accel=on, auto means the value is automatically derived from the host SMMU. + - With accel=off, auto is resolved to 'off'. + ``-device amd-iommu[,option=...]`` Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU). Only available with ``-machine q35``, it supports the following options:
Signature · 1 line
2.43.0
Quoted 2 lines · Shameer Kolothum
Please find the complete branch here:
https://github.com/shamiali2008/qemu-master/tree/master-vcmdq-v6-ext
Sanity on my side looks good: