Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD372D41D6B for ; Thu, 11 Dec 2025 17:55:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vTksV-0006Yt-Kl; Thu, 11 Dec 2025 12:55:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vTksU-0006Yj-OE for qemu-arm@nongnu.org; Thu, 11 Dec 2025 12:54:58 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vTksS-0002pd-TB for qemu-arm@nongnu.org; Thu, 11 Dec 2025 12:54:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1765475695; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OSfi5Q2D1Ez0ezrQYwKOOwbqGJVPaX20e4ez2ep3TCk=; b=TxuWafXzANlzcbqCnj871BE88IX/s8j6zuewnENHzlg0TF6ZaGh5PYGM46CkntxnfH9jPa hl6mjue4FOr9ZPsowo2CDGkPYvd733CzO8Yu7ByKaFW9aG9a8cbm1At0so7S0EY9sMJTvf U44a+p/aETojvPWR4iJ4Dat0qDOwne0= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-672-SCpdodAYM6qhSAczxWw_gw-1; Thu, 11 Dec 2025 12:54:54 -0500 X-MC-Unique: SCpdodAYM6qhSAczxWw_gw-1 X-Mimecast-MFC-AGG-ID: SCpdodAYM6qhSAczxWw_gw_1765475694 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-8b24383b680so107163985a.0 for ; Thu, 11 Dec 2025 09:54:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765475694; x=1766080494; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:reply-to:user-agent:mime-version:date :message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OSfi5Q2D1Ez0ezrQYwKOOwbqGJVPaX20e4ez2ep3TCk=; b=uew4hNRwVcc3TPUD7wy+f64Vlyd1xf2NNgiuYA0YL7zLtU4MxsuUnG3IMqoXkFBPWd 3Q4pTCXoUSwecZ9rKibowYILb4HCN2DFQfDVY4YukuvznKjIL54Mn1VlXjMXx6OWwzzj xjzjlf/thYheA+cSEp5+6wcIS4Mwjjnkxvzx5OkAF2LqyOHN5og2rOdpJFErN/r9cOSA D5NN94uSsEirk0KfBkMwRjcFhhL27fE0uD7WPjdL6o7BalLq4XhMLLn7g36QHR3X/l0X MiHEcnoJurCepkgWbAZeI1e2CbCAfrfLBJiZ7Zj/Pt27XZ9/F3A93DkLRpEbxkAeJ8n3 pHng== X-Forwarded-Encrypted: i=1; AJvYcCWXzcJuD5I88JdXwYe0+EKyrpE2j19ayYhFS3d4cx03OIpKLipyzI4Hhnq7iuVcPH2l+NXTs+ym1w==@nongnu.org X-Gm-Message-State: AOJu0YyHC7TI3wSCsUKECPnKTNP84iqXtdytqzDSmC4a3rxl2SKZ/SB9 qYf3u6HKHCd9oUxuXJnCJN6xVsfoFk+GnslY4EktXzUj9XZuEsH3iUN6Z6u/r1ZEIOWpzV4NO3U u65mNAzrhnaXFgRXyGYCKbIKjbbvS1qgZmLSBWuq49XTL3qfJYo+KIQ== X-Gm-Gg: AY/fxX5G+xd37aLzmZSPViNCdxl/djHtSxuws4+T8k7Zt37KPZjTx5tWe0Ao+JaW5Yr CyXgV0OluXqzSrmhv0hgfPVeftsvj9iRstoJT+7NMf0g3maWd3xKYm4GvlrjzySyCA755nm2R/s Tn5m8p4gew8prt40HekNf9yjNzgr8AissDMr0gJkAAuiV48yyxfnDesNXOZB921igR/OjDX94/Z DzA/Kh2t+pKsD1UjOd3DU5o8HYL0xox1fh9NmyBJViHuWwwlLsf8Pw7LY9SUKIRG85VQW2BZ1/y +4UevLQelk+OuVuagR7xM8FyW4Y+w9QvNGV240R5BLkThhOC1AaMRe14z2brAU8SgDAciNntIyG VNiLSM6OwGyDg5hl98OadZENuLnCruZTtG3Ncb+58q7hqF+1bxz+CwMQJiw== X-Received: by 2002:a05:620a:cfc:b0:8b2:efe7:d82b with SMTP id af79cd13be357-8ba38c77fa0mr691202185a.30.1765475694001; Thu, 11 Dec 2025 09:54:54 -0800 (PST) X-Google-Smtp-Source: AGHT+IGqvjYfoHc2YGk8W6Q0S+7Izbcr6XIU/YaX/Haz/WiuWhDguSVusaCMTdcGFz9FeWKKoVhqFg== X-Received: by 2002:a05:620a:cfc:b0:8b2:efe7:d82b with SMTP id af79cd13be357-8ba38c77fa0mr691199185a.30.1765475693585; Thu, 11 Dec 2025 09:54:53 -0800 (PST) Received: from ?IPV6:2a01:e0a:f0e:9070:527b:9dff:feef:3874? ([2a01:e0a:f0e:9070:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8bab544b657sm269021785a.10.2025.12.11.09.54.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 11 Dec 2025 09:54:53 -0800 (PST) Message-ID: Date: Thu, 11 Dec 2025 18:54:50 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH 00/16] hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3 To: Shameer Kolothum , qemu-arm@nongnu.org, qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, nicolinc@nvidia.com, nathanc@nvidia.com, mochs@nvidia.com, jgg@nvidia.com, jonathan.cameron@huawei.com, zhangfei.gao@linaro.org, zhenzhong.duan@intel.com, kjaju@nvidia.com References: <20251210133737.78257-1-skolothumtho@nvidia.com> From: Eric Auger In-Reply-To: <20251210133737.78257-1-skolothumtho@nvidia.com> X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Iaxg3oJYlta_VbxsDuDjNhH6pcAP964p9M_X27qAjGQ_1765475694 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=170.10.133.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: eric.auger@redhat.com Errors-To: qemu-arm-bounces+qemu-arm=archiver.kernel.org@nongnu.org Sender: qemu-arm-bounces+qemu-arm=archiver.kernel.org@nongnu.org Hi Shameer, On 12/10/25 2:37 PM, Shameer Kolothum wrote: > Hi, > > This RFC series adds initial support for NVIDIA Tegra241 CMDQV > (Command Queue Virtualisation), an extension to ARM SMMUv3 that > provides hardware accelerated virtual command queues (VCMDQs) for > guests. CMDQV allows guests to issue SMMU invalidation commands > directly to hardware without VM exits, significantly reducing TLBI > overhead. > > Thanks to Nicolin for the initial patches and testing on which this RFC > is based. > > This is based on v6[0] of the SMMUv3 accel series, which is still under > review, though nearing convergence. This is sent as an RFC, with the goal > of gathering early feedback on the CMDQV design and its integration with > the SMMUv3 acceleration path. > > Background: > > Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" > (VINTFs), each hosting up to 128 VCMDQs. > > Each VINTF exposes two 64KB MMIO pages: > - Page0 – guest owned control and status registers (directly mapped > into the VM) > - Page1 – queue configuration registers (trapped/emulated by QEMU) > > Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not > support the full command set. Only a subset, primarily invalidation > related commands, is accepted by the CMDQV hardware. For this reason, > a distinct CMDQV device must be exposed to the guest, and the guest OS > must include a Tegra241 CMDQV aware driver to take advantage of the > hardware acceleration. > > VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, > allowing QEMU to attach guest configured VCMDQ buffers to the > underlying CMDQV hardware through IOMMUFD. The Linux kernel already > supports the full CMDQV virtualisation model via IOMMUFD[0]. > > Summary of QEMU changes: > > - Integrated into the existing SMMUv3 accel path via a > "tegra241-cmdqv" property. > - Support for allocating vIOMMU objects of type > IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV. > - Mapping and emulation of the CMDQV MMIO register layout. > - VCMDQ/VINTF read/write handling and queue allocation using IOMMUFD > APIs. > - Reset and initialisation hooks, including checks for at least one > cold-plugged device. > - CMDQV hardware reads guest queue memory using host physical addresses > provided through IOMMUFD, which requires that the VCMDQ buffer be > physically contiguous not only in guest PA space but also in host > PA space. When Tegra241 CMDQV is enabled, QEMU must therefore only > expose a CMDQV size that the host can reliably back with contiguous > physical memory. Because of this constraint, it is suggested to use > huge pages to back the guest RAM. > - ACPI DSDT node generation for CMDQV devices on the virt machine. > > These patches have been sanity tested on NVIDIA Grace platforms. > > ToDo / revisit: > - Prevent hot-unplug of the last device associated with vIOMMU as > this might allow associating a different host SMMU/CMDQV. > - Locking requirements around error event propagation. > > Feedback and testing are very welcome. > > Thanks, > Shameer > [0] https://lore.kernel.org/qemu-devel/20251120132213.56581-1-skolothumtho@nvidia.com/ > [1] https://lore.kernel.org/all/cover.1752126748.git.nicolinc@nvidia.com/ do you have a branch to share with all the bits? Thanks Eric > > Nicolin Chen (12): > backends/iommufd: Update iommufd_backend_get_device_info > backends/iommufd: Update iommufd_backend_alloc_viommu to allow user > ptr > backends/iommufd: Introduce iommufd_backend_alloc_hw_queue > backends/iommufd: Introduce iommufd_backend_viommu_mmap > hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation > support > hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest > hw/arm/tegra241-cmdqv: Add read emulation support for registers > system/physmem: Add helper to check whether a guest PA maps to RAM > hw/arm/tegra241-cmdqv:: Add write emulation for registers > hw/arm/tegra241-cmdqv: Add reset handler > hw/arm/tegra241-cmdqv: Limit queue size based on backend page size > hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT > > Shameer Kolothum (4): > hw/arm/tegra241-cmdqv: Allocate vEVENTQ object > hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors > virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev > hw/arm/smmuv3: Add tegra241-cmdqv property for SMMUv3 device > > backends/iommufd.c | 65 ++++ > backends/trace-events | 2 + > hw/arm/Kconfig | 5 + > hw/arm/meson.build | 1 + > hw/arm/smmuv3-accel.c | 16 +- > hw/arm/smmuv3.c | 18 + > hw/arm/tegra241-cmdqv.c | 759 ++++++++++++++++++++++++++++++++++++++ > hw/arm/tegra241-cmdqv.h | 337 +++++++++++++++++ > hw/arm/trace-events | 5 + > hw/arm/virt-acpi-build.c | 110 +++++- > hw/vfio/iommufd.c | 6 +- > include/exec/cpu-common.h | 2 + > include/hw/arm/smmuv3.h | 3 + > include/hw/arm/virt.h | 2 + > include/system/iommufd.h | 16 + > system/physmem.c | 12 + > 16 files changed, 1332 insertions(+), 27 deletions(-) > create mode 100644 hw/arm/tegra241-cmdqv.c > create mode 100644 hw/arm/tegra241-cmdqv.h >