hw/arm/smmuv3-accel: Resolve AUTO properties
v6 cover hw/arm/smmuv3-accel: Resolve AUTO properties
Hi,
This is a follow-up to the series [0] that introduces support for resolving 'auto' for arm-smmuv3 accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties based on host IOMMU capabilities. This is dependent on the series [1] for changing these property types to accept 'auto' values.
Accelerated SMMUv3 Address Translation Services support is derived from IDR0, Range Invalidation support is derived from IDR3, Substream ID size is derived from IDR1, and output address space is derived from IDR5.
The default values are set to 'auto' for all properties. If accel=off and the values are set to 'auto' or are omitted and resolve to 'auto', the default property values defined in smmuv3_init_id_regs() for OAS and RIL will remain unchanged, while SSIDSIZE and ATS values will remain initialized at 0.
A complete branch can be found here:
https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto-resolve-v6
Please take a look and let me know your feedback.
Thanks,
Nathan
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/qemu-arm/[email protected]/
Changes from v5:
- Added R-by tags, thanks all!
- Rebase over latest upstream
(https://lore.kernel.org/all/[email protected]/)
Changes from v4:
- Added R-by tags, thanks all!
- Re-word smmuv3_class_init() accel description to list nested translation support as a pre-requisite
- Fix typo in smmuv3_class_init() SubstreamID size description
Changes from v3:
- Added R-by tags, thanks all!
- Include a fix patch for resolving ATS support in ATC invalidation case
- Update host requirements documented for accel=on, oas, and ssidsize
- Fix style issue, update comments
- Update function name for smmuv3_accel_machine_done
- Remove unnecessary early return for smmuv3_accel_machine_done
- Fix formatting in qemu-options.hx based on html render
Changes from v2:
- Added R-by tags, thanks Eric, Shameer, Cédric for the feedback!
- Re-worded descriptions in qemu-options.hx and smmu_validate_property()
- Emphasize accel=off auto behavior in a comment for smmuv3_properties[]
Changes from v1:
- Update qemu-options.hx to explicitly mention auto is supported
- Update qemu-options.hx to mention auto as default
- Simplify comments in smmuv3_accel_auto_finalise()
- Update descriptions in smmuv3_class_init()
- Consolidate all default value change patches into one patch
- Implement and use smmuv3_pasid_supported() for setting PASID support flag
- Move smmuv3_machine_done() and qemu_add_machine_init_done_notifier() to smmuv3-accel.c
- Register machine done notifier only in case auto_mode is set
Testing:
Basic sanity testing was performed on an NVIDIA Grace platform with GPU device assignment and running CUDA test apps on the guest. Observed the feature properties being set based on host IOMMU capabilities. Verified that the VM boot will fail without a cold-plugged device, and that a hot-plugged device re-uses the resolved values from the initial cold-plug. Additional testing and feedback are welcome.
Nathan Chen (9):
hw/arm/smmuv3: Update ATC invalidation check hw/arm/smmuv3: Improve accel SMMUv3 usage documentation hw/arm/smmuv3-accel: Add helper for resolving auto parameters hw/arm/smmuv3-accel: Implement "auto" value for "ats" hw/arm/smmuv3-accel: Implement "auto" value for "ril" hw/arm/smmuv3-accel: Implement "auto" value for "ssidsize" hw/arm/smmuv3-accel: Implement "auto" value for "oas" hw/arm/smmuv3: Set default ats, ril, ssidsize, oas to auto qemu-options.hx: Support "auto" for accel SMMUv3 properties
hw/arm/smmuv3-accel.c | 77 ++++++++++++++++++++++++++++++++++++++- hw/arm/smmuv3-accel.h | 2 + hw/arm/smmuv3.c | 81 +++++++++++++++++++++++------------------ hw/core/machine.c | 5 +++ include/hw/arm/smmuv3.h | 3 ++ qemu-options.hx | 25 ++++++++++--- 6 files changed, 150 insertions(+), 43 deletions(-)
Signature · 1 line
2.43.0
PATCH v6 1/9 hw/arm/smmuv3: Update ATC invalidation check
From: Nathan Chen <[email protected]>
Use smmuv3_ats_enabled() to determine whether ATS is enabled for the guest when handling an ATC invalidation command, as setting the ATS property value to 'auto' will resolve to ATS being detected as enabled in the ATC invalidation check otherwise.
--- hw/arm/smmuv3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/arm/smmuv3.c +1 -1
@@ -1528,7 +1528,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, Error **errp) {SMMUDevice *sdev =smmu_find_sdev (bs,CMD_SID (&cmd)); -if (!sdev || !s->ats) { +if (!sdev || !smmuv3_ats_enabled (s)) {trace_smmuv3_unhandled_cmd (type);break ; }
Signature · 1 line
2.43.0
PATCH v6 2/9 hw/arm/smmuv3: Improve accel SMMUv3 usage documentation
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Add a statement to clarify that the host SMMUv3 must support HW-accelerated vfio-pci device assignment when setting accel=on.
--- hw/arm/smmuv3.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
hw/arm/smmuv3.c +3 -1
@@ -2165,7 +2165,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)object_class_property_set_description (klass,"accel" ,"Enable SMMUv3 accelerator support. Allows host SMMUv3 to be " -"configured in nested mode for vfio-pci dev assignment" ); +"configured in nested mode for vfio-pci dev assignment. Please " +"ensure the host SMMUv3 supports nested translation before " +"enabling." );object_class_property_set_description (klass,"ril" ,"Disable range invalidation support (for accel=on). ril=auto " "is not supported." );
Signature · 1 line
2.43.0
RE: [PATCH v6 2/9] hw/arm/smmuv3: Improve accel SMMUv3 usage documentation
Quoted 15 lines
-----Original Message-----
From: Nathan Chen <[email protected]>
Sent: 08 June 2026 18:49
To: [email protected]; [email protected]
Cc: Eric Auger <[email protected]>; Peter Maydell <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Zhao Liu <[email protected]>; Matt Ochs <[email protected]>; Nicolin Chen <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Nathan Chen <[email protected]>; qemu- [email protected]
Subject: [PATCH v6 2/9] hw/arm/smmuv3: Improve accel SMMUv3 usage documentation
Quoted 8 lines
From: Nathan Chen <[email protected]>
Add a statement to clarify that the host SMMUv3 must support HW- accelerated vfio-pci device assignment when setting accel=on.
Reported-by Eric Auger <[email protected]>Reviewed-by Eric Auger <[email protected]>Signed-off-by Nathan Chen <[email protected]>
Thanks,
Shameer
PATCH v6 3/9 hw/arm/smmuv3-accel: Add helper for resolving auto parameters
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Introduce smmuv3_accel_auto_finalise() to resolve properties that are set to 'auto' for accelerated SMMUv3. This helper function allows properties such as ats, ril, ssidsize, and oas support to be resolved from host IOMMU capabilities via IOMMU_GET_HW_INFO.
The later commits in this series set the auto_mode flag to true when an accel SMMUv3 property value is explicitly set to 'auto', or if the property value is not set and defaults to auto mode.
Setting these property values to 'auto' requires at least one cold-plugged device to retrieve and finalise these properties. If the auto_mode flag is true, register a machine_init_done notifier to verify this requirement and fail boot if it is not met.
Hot-plugged devices into an accel SMMUv3-associated bus will re-use the resolved host values from the initial cold-plug.
Subsequent patches will make use of this helper to resolve 'auto' to what is reported by host IOMMU capabilities.
--- hw/arm/smmuv3-accel.c | 41 +++++++++++++++++++++++++++++++++++++++++ hw/arm/smmuv3-accel.h | 2 ++ include/hw/arm/smmuv3.h | 3 +++ 3 files changed, 46 insertions(+)
hw/arm/smmuv3-accel.c +41 -0
@@ -18,6 +18,7 @@#include "smmuv3-internal.h" #include "smmuv3-accel.h" +#include "system/system.h" /* * The root region aliases the global system memory, and shared_as_sysmem
@@ -35,11 +36,33 @@ static int smmuv3_oas_bits(uint32_t oas)return map[oas]; } +static void smmuv3_accel_auto_finalise (SMMUv3State *s, +struct iommu_hw_info_arm_smmuv3 *info) +{ +SMMUv3AccelState *accel = s->s_accel; + +/* +* Return if 'auto' was not set for any accel SMMUv3 property, or +* if property values were already resolved from a previous call +* to this function (e.g. if this function was called again after +* VM boot during device hot plug). We do not accept new property +* values in this case where auto_finalised == true, and we re-use +* the values determined from the initial cold plug. +*/ +if (!accel->auto_mode || accel->auto_finalised) { +return ; + } + + accel->auto_finalised = true; +} +static bool smmuv3_accel_check_hw_compatible (SMMUv3State *s,struct iommu_hw_info_arm_smmuv3 *info,Error **errp) { +smmuv3_accel_auto_finalise (s, info); +/* QEMU SMMUv3 supports both linear and 2-level stream tables */ if (FIELD_EX32 (info->idr[0 ],IDR0 ,STLEVEL ) !=FIELD_EX32 (s->idr[0 ],IDR0 ,STLEVEL )) {
@@ -918,6 +941,18 @@ static void smmuv3_accel_as_init(SMMUv3State *s)address_space_init (shared_as_sysmem, &root,"smmuv3-accel-as-sysmem" ); } +static void smmuv3_accel_machine_done (Notifier *notifier,void *data) +{ +SMMUv3State *s =container_of (notifier, SMMUv3State, machine_done); +SMMUv3AccelState *accel = s->s_accel; + +if (accel->auto_mode && !accel->auto_finalised) { +error_report ("arm-smmuv3 accel=on with 'auto' properties requires " +"at least one cold-plugged VFIO device" ); +exit (1 ); + } +} +bool smmuv3_accel_init (SMMUv3State *s,Error **errp) {SMMUState *bs =ARM_SMMU (s);
@@ -925,5 +960,11 @@ bool smmuv3_accel_init(SMMUv3State *s, Error **errp) s->s_accel =g_new0 (SMMUv3AccelState,1 ); bs->iommu_ops = &smmuv3_accel_ops;smmuv3_accel_as_init (s); + +if (s->s_accel->auto_mode) { + s->machine_done.notify = smmuv3_accel_machine_done; +qemu_add_machine_init_done_notifier (&s->machine_done); + } +return true; }
hw/arm/smmuv3-accel.h +2 -0
@@ -25,6 +25,8 @@ typedef struct SMMUv3AccelState {uint32_t bypass_hwpt_id;uint32_t abort_hwpt_id; QLIST_HEAD(, SMMUv3AccelDevice) device_list; +bool auto_mode; +bool auto_finalised; } SMMUv3AccelState; typedef struct SMMUS1Hwpt {
include/hw/arm/smmuv3.h +3 -0
@@ -22,6 +22,7 @@#include "hw/arm/smmu-common.h" #include "qom/object.h" #include "qapi/qapi-types-misc-arm.h" +#include "qemu/notify.h" #define TYPE_SMMUV3_IOMMU_MEMORY_REGION "smmuv3-iommu-memory-region"
@@ -74,6 +75,8 @@ struct SMMUv3State {OnOffAuto ats;OasMode oas;SsidSizeMode ssidsize; + +Notifier machine_done; }; typedef enum {
Signature · 1 line
2.43.0
PATCH v6 4/9 hw/arm/smmuv3-accel: Implement "auto" value for "ats"
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Allow accelerated SMMUv3 Address Translation Services support property to be derived from host IOMMU capabilities. Derive host values using IOMMU_GET_HW_INFO, retrieving ATS capability from IDR0.
--- hw/arm/smmuv3-accel.c | 9 +++++++++ hw/arm/smmuv3.c | 11 ++++------- 2 files changed, 13 insertions(+), 7 deletions(-)
hw/arm/smmuv3-accel.c +9 -0
@@ -53,6 +53,11 @@ static void smmuv3_accel_auto_finalise(SMMUv3State *s,return ; } +if (s->ats ==ON_OFF_AUTO_AUTO ) { + s->idr[0 ] =FIELD_DP32 (s->idr[0 ],IDR0 ,ATS , +FIELD_EX32 (info->idr[0 ],IDR0 ,ATS )); + } + accel->auto_finalised = true; }
@@ -961,6 +966,10 @@ bool smmuv3_accel_init(SMMUv3State *s, Error **errp) bs->iommu_ops = &smmuv3_accel_ops;smmuv3_accel_as_init (s); +if (s->ats ==ON_OFF_AUTO_AUTO ) { + s->s_accel->auto_mode = true; + } +if (s->s_accel->auto_mode) { s->machine_done.notify = smmuv3_accel_machine_done;qemu_add_machine_init_done_notifier (&s->machine_done);
hw/arm/smmuv3.c +4 -7
@@ -1965,10 +1965,6 @@ static void smmu_reset_exit(Object *obj, ResetType type)static bool smmu_validate_property (SMMUv3State *s,Error **errp) { -if (s->ats ==ON_OFF_AUTO_AUTO ) { -error_setg (errp,"ats auto mode is not supported" ); -return false; - }if (s->ril ==ON_OFF_AUTO_AUTO ) {error_setg (errp,"ril auto mode is not supported" );return false;
@@ -2172,9 +2168,10 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"Disable range invalidation support (for accel=on). ril=auto " "is not supported.");object_class_property_set_description (klass,"ats" , -"Enable/disable ATS support (for accel=on). Please ensure host " -"platform has ATS support before enabling this. ats=auto is not " -"supported." ); +"Enable/disable ATS support (for accel=on). " +"Valid values are on, off, and auto. Defaults to off. " +"Please ensure host platform supports ATS before setting it " +"to on." );object_class_property_set_description (klass,"oas" ,"Specify Output Address Size (for accel=on). Supported values " "are 44 or 48 bits. Defaults to 44 bits. oas=auto is not "
Signature · 1 line
2.43.0
PATCH v6 5/9 hw/arm/smmuv3-accel: Implement "auto" value for "ril"
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Allow accelerated SMMUv3 Range Invalidation support property to be derived from host IOMMU capabilities. Derive host values using IOMMU_GET_HW_INFO, retrieving RIL capability from IDR3.
--- hw/arm/smmuv3-accel.c | 8 +++++++- hw/arm/smmuv3.c | 10 ++++------ 2 files changed, 11 insertions(+), 7 deletions(-)
hw/arm/smmuv3-accel.c +7 -1
@@ -58,6 +58,11 @@ static void smmuv3_accel_auto_finalise(SMMUv3State *s, FIELD_EX32(info->idr[0], IDR0, ATS)); } +if (s->ril ==ON_OFF_AUTO_AUTO ) { + s->idr[3 ] =FIELD_DP32 (s->idr[3 ],IDR3 ,RIL , +FIELD_EX32 (info->idr[3 ],IDR3 ,RIL )); + } + accel->auto_finalised = true; }
@@ -966,7 +971,8 @@ bool smmuv3_accel_init(SMMUv3State *s, Error **errp) bs->iommu_ops = &smmuv3_accel_ops;smmuv3_accel_as_init (s); -if (s->ats ==ON_OFF_AUTO_AUTO ) { +if (s->ats ==ON_OFF_AUTO_AUTO || + s->ril ==ON_OFF_AUTO_AUTO ) { s->s_accel->auto_mode = true; }
hw/arm/smmuv3.c +4 -6
@@ -1965,10 +1965,6 @@ static void smmu_reset_exit(Object *obj, ResetType type)static bool smmu_validate_property (SMMUv3State *s,Error **errp) { -if (s->ril ==ON_OFF_AUTO_AUTO ) { -error_setg (errp,"ril auto mode is not supported" ); -return false; - }if (s->ssidsize ==SSID_SIZE_MODE_AUTO ) {error_setg (errp,"ssidsize auto mode is not supported" );return false;
@@ -2165,8 +2161,10 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"ensure the host SMMUv3 supports nested translation before " "enabling.");object_class_property_set_description (klass,"ril" , -"Disable range invalidation support (for accel=on). ril=auto " -"is not supported." ); +"Enable/disable range invalidation support (for accel=on). " +"Valid values are on, off, and auto. Defaults to on. " +"Any attempt to turn it 'on' while the host does not support " +"it would fail." );object_class_property_set_description (klass,"ats" ,"Enable/disable ATS support (for accel=on). " "Valid values are on, off, and auto. Defaults to off. "
Signature · 1 line
2.43.0
PATCH v6 6/9 hw/arm/smmuv3-accel: Implement "auto" value for "ssidsize"
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Allow accelerated SMMUv3 SSID size property to be derived from host IOMMU capabilities. Derive host values using IOMMU_GET_HW_INFO, retrieving SSID size from IDR1. When the auto SSID size is resolved to a non-zero value, PASID capability is advertised to the vIOMMU and accelerated use cases such as Shared Virtual Addressing (SVA) are supported.
--- hw/arm/smmuv3-accel.c | 17 +++++++++++++++-- hw/arm/smmuv3.c | 22 ++++++++++++---------- 2 files changed, 27 insertions(+), 12 deletions(-)
hw/arm/smmuv3-accel.c +15 -2
@@ -63,6 +63,11 @@ static void smmuv3_accel_auto_finalise(SMMUv3State *s, FIELD_EX32(info->idr[3], IDR3, RIL)); } +if (s->ssidsize ==SSID_SIZE_MODE_AUTO ) { + s->idr[1 ] =FIELD_DP32 (s->idr[1 ],IDR1 ,SSIDSIZE , +FIELD_EX32 (info->idr[1 ],IDR1 ,SSIDSIZE )); + } + accel->auto_finalised = true; }
@@ -823,6 +828,13 @@ static AddressSpace *smmuv3_accel_find_add_as(PCIBus *bus, void *opaque, } } +static inline bool smmuv3_pasid_supported (SMMUv3State *s) +{ +return s->ssidsize >SSID_SIZE_MODE_0 || + (s->ssidsize ==SSID_SIZE_MODE_AUTO && +FIELD_EX32 (s->idr[1 ],IDR1 ,SSIDSIZE )); +} +static uint64_t smmuv3_accel_get_viommu_flags (void *opaque) {/*
@@ -835,7 +847,7 @@ static uint64_t smmuv3_accel_get_viommu_flags(void *opaque)SMMUState *bs = opaque;SMMUv3State *s =ARM_SMMUV3 (bs); -if (s->ssidsize >SSID_SIZE_MODE_0 ) { +if (smmuv3_pasid_supported (s)) { flags |=VIOMMU_FLAG_PASID_SUPPORTED ; }return flags;
@@ -972,7 +984,8 @@ bool smmuv3_accel_init(SMMUv3State *s, Error **errp)smmuv3_accel_as_init (s);if (s->ats ==ON_OFF_AUTO_AUTO || - s->ril ==ON_OFF_AUTO_AUTO ) { + s->ril ==ON_OFF_AUTO_AUTO || + s->ssidsize ==SSID_SIZE_MODE_AUTO ) { s->s_accel->auto_mode = true; }
hw/arm/smmuv3.c +12 -10
@@ -626,7 +626,10 @@ static int decode_ste(SMMUv3State *s, SMMUTransCfg *cfg, }/* Multiple context descriptors require SubstreamID support */ -if (s->ssidsize ==SSID_SIZE_MODE_0 &&STE_S1CDMAX (ste) !=0 ) { +if ((s->ssidsize ==SSID_SIZE_MODE_0 || + (s->ssidsize ==SSID_SIZE_MODE_AUTO && + !FIELD_EX32 (s->idr[1 ],IDR1 ,SSIDSIZE ))) && +STE_S1CDMAX (ste) !=0 ) {qemu_log_mask (LOG_UNIMP ,"SMMUv3: multiple S1 context descriptors require SubstreamID support. " "Configure ssidsize > 0 (requires accel=on)\n" );
@@ -1965,10 +1968,6 @@ static void smmu_reset_exit(Object *obj, ResetType type)static bool smmu_validate_property (SMMUv3State *s,Error **errp) { -if (s->ssidsize ==SSID_SIZE_MODE_AUTO ) { -error_setg (errp,"ssidsize auto mode is not supported" ); -return false; - }if (s->oas !=OAS_MODE_44 && s->oas !=OAS_MODE_48 ) {error_setg (errp,"QEMU SMMUv3 model only implements 44 and 48 bit" "OAS; other OasMode values are not supported" );
@@ -1989,7 +1988,8 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)return false; }if (s->ssidsize >SSID_SIZE_MODE_0 ) { -error_setg (errp,"ssidsize can only be set if accel=on" ); +error_setg (errp,"ssidsize can only be greater than 0 " +"bits if accel=on" );return false; }return true;
@@ -2175,11 +2175,13 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"are 44 or 48 bits. Defaults to 44 bits. oas=auto is not " "supported.");object_class_property_set_description (klass,"ssidsize" , -"Number of bits used to represent SubstreamIDs (SSIDs). " +"Set number of bits used to represent SubstreamIDs (SSIDs). " +"Valid values are 0-20 and auto. Defaults to 0. " "A value of N allows SSIDs in the range [0 .. 2^N - 1]. " -"Valid range is 0-20, where 0 disables SubstreamID support. " -"Defaults to 0. A value greater than 0 is required to enable " -"PASID support. ssidsize=auto is not supported." ); +"A value of 0 disables SubstreamID support. A value greater " +"than 0 is required to enable PASID support." +"Please ensure the value does not exceed the maximum " +"SubstreamID size supported by the host platform." ); } static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
Signature · 1 line
2.43.0
PATCH v6 7/9 hw/arm/smmuv3-accel: Implement "auto" value for "oas"
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Allow accelerated SMMUv3 OAS property to be derived from host IOMMU capabilities. Derive host values using IOMMU_GET_HW_INFO, retrieving OAS from IDR5.
This keeps the OAS value advertised by the virtual SMMU compatible with the capabilities of the host SMMUv3, so that the intermediate physical addresses (IPA) consumed by host SMMU for stage-2 translation do not exceed the host's max supported IPA size.
--- hw/arm/smmuv3-accel.c | 8 +++++++- hw/arm/smmuv3.c | 17 ++++++++++------- 2 files changed, 17 insertions(+), 8 deletions(-)
hw/arm/smmuv3-accel.c +7 -1
@@ -68,6 +68,11 @@ static void smmuv3_accel_auto_finalise(SMMUv3State *s, FIELD_EX32(info->idr[1], IDR1, SSIDSIZE)); } +if (s->oas ==OAS_MODE_AUTO ) { + s->idr[5 ] =FIELD_DP32 (s->idr[5 ],IDR5 ,OAS , +FIELD_EX32 (info->idr[5 ],IDR5 ,OAS )); + } + accel->auto_finalised = true; }
@@ -985,7 +990,8 @@ bool smmuv3_accel_init(SMMUv3State *s, Error **errp)if (s->ats ==ON_OFF_AUTO_AUTO || s->ril ==ON_OFF_AUTO_AUTO || - s->ssidsize ==SSID_SIZE_MODE_AUTO ) { + s->ssidsize ==SSID_SIZE_MODE_AUTO || + s->oas ==OAS_MODE_AUTO ) { s->s_accel->auto_mode = true; }
hw/arm/smmuv3.c +10 -7
@@ -1968,9 +1968,11 @@ static void smmu_reset_exit(Object *obj, ResetType type)static bool smmu_validate_property (SMMUv3State *s,Error **errp) { -if (s->oas !=OAS_MODE_44 && s->oas !=OAS_MODE_48 ) { -error_setg (errp,"QEMU SMMUv3 model only implements 44 and 48 bit" -"OAS; other OasMode values are not supported" ); +if (s->oas !=OAS_MODE_44 && s->oas !=OAS_MODE_48 && + s->oas !=OAS_MODE_AUTO ) { +error_setg (errp,"QEMU SMMUv3 model only implements auto, " +"44 bit, or 48 bit OAS. Other OasMode values are " +"not supported." );return false; }
@@ -1984,7 +1986,7 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)return false; }if (s->oas >OAS_MODE_44 ) { -error_setg (errp,"OAS must be 44 bits when accel=off" ); +error_setg (errp,"oas must be 44 bits when accel=off" );return false; }if (s->ssidsize >SSID_SIZE_MODE_0 ) {
@@ -2171,9 +2173,10 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)"Please ensure host platform supports ATS before setting it " "to on.");object_class_property_set_description (klass,"oas" , -"Specify Output Address Size (for accel=on). Supported values " -"are 44 or 48 bits. Defaults to 44 bits. oas=auto is not " -"supported." ); +"Set Output Address Size in bits (for accel=on). " +"Valid values are 44, 48, and auto. Defaults to 44 bits." +"Please ensure the value does not exceed the maximum " +"Output Address Size supported by the host platform." );object_class_property_set_description (klass,"ssidsize" ,"Set number of bits used to represent SubstreamIDs (SSIDs). " "Valid values are 0-20 and auto. Defaults to 0. "
Signature · 1 line
2.43.0
PATCH v6 8/9 hw/arm/smmuv3: Set default ats, ril, ssidsize, oas to auto
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Set the default value of ATS, RIL, SSIDSIZE, and OAS to auto, in order to match the host IOMMU properties when accel=on.
If accel=off and these property values are set to auto, the default property values defined in smmuv3_init_id_regs() for OAS and RIL will remain unchanged, while SSIDSIZE and ATS values will remain initialized at 0.
Introduce a new compat for the changed defaults.
--- hw/arm/smmuv3.c | 23 +++++++++++++++-------- hw/core/machine.c | 5 +++++ 2 files changed, 20 insertions(+), 8 deletions(-)
hw/arm/smmuv3.c +15 -8
@@ -2129,12 +2129,19 @@ static const Property smmuv3_properties[] = {DEFINE_PROP_BOOL ("accel" , SMMUv3State, accel, false),/* GPA of MSI doorbell, for SMMUv3 accel use. */ DEFINE_PROP_UINT64 ("msi-gpa" , SMMUv3State, msi_gpa,0 ), +/* +* AUTO values for accel=off will resolve to: +* ril: on +* ats: off +* oas: 44 +* ssidsize: 0 +*/ /* RIL can be turned off for accel cases */ -DEFINE_PROP_ON_OFF_AUTO ("ril" , SMMUv3State, ril,ON_OFF_AUTO_ON ), -DEFINE_PROP_ON_OFF_AUTO ("ats" , SMMUv3State, ats,ON_OFF_AUTO_OFF ), -DEFINE_PROP_OAS_MODE ("oas" , SMMUv3State, oas,OAS_MODE_44 ), +DEFINE_PROP_ON_OFF_AUTO ("ril" , SMMUv3State, ril,ON_OFF_AUTO_AUTO ), +DEFINE_PROP_ON_OFF_AUTO ("ats" , SMMUv3State, ats,ON_OFF_AUTO_AUTO ), +DEFINE_PROP_OAS_MODE ("oas" , SMMUv3State, oas,OAS_MODE_AUTO ),DEFINE_PROP_SSIDSIZE_MODE ("ssidsize" , SMMUv3State, ssidsize, - SSID_SIZE_MODE_0), + SSID_SIZE_MODE_AUTO), };static void smmuv3_instance_init (Object *obj)
@@ -2164,22 +2171,22 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data) "enabling.");object_class_property_set_description (klass,"ril" ,"Enable/disable range invalidation support (for accel=on). " -"Valid values are on, off, and auto. Defaults to on. " +"Valid values are on, off, and auto. Defaults to auto. " "Any attempt to turn it 'on' while the host does not support " "it would fail." );object_class_property_set_description (klass,"ats" ,"Enable/disable ATS support (for accel=on). " -"Valid values are on, off, and auto. Defaults to off. " +"Valid values are on, off, and auto. Defaults to auto. " "Please ensure host platform supports ATS before setting it " "to on." );object_class_property_set_description (klass,"oas" ,"Set Output Address Size in bits (for accel=on). " -"Valid values are 44, 48, and auto. Defaults to 44 bits." +"Valid values are 44, 48, and auto. Defaults to auto." "Please ensure the value does not exceed the maximum " "Output Address Size supported by the host platform." );object_class_property_set_description (klass,"ssidsize" ,"Set number of bits used to represent SubstreamIDs (SSIDs). " -"Valid values are 0-20 and auto. Defaults to 0. " +"Valid values are 0-20 and auto. Defaults to auto. " "A value of N allows SSIDs in the range [0 .. 2^N - 1]. " "A value of 0 disables SubstreamID support. A value greater " "than 0 is required to enable PASID support."
hw/core/machine.c +5 -0
@@ -38,11 +38,16 @@#include "hw/virtio/virtio-iommu.h" #include "hw/acpi/generic_event_device.h" #include "qemu/audio.h" +#include "hw/arm/smmuv3.h" GlobalProperty hw_compat_11_0[] = { {"chardev-vc" ,"encoding" ,"cp437" }, {"tpm-crb" ,"cap-chunk" ,"off" }, {"tpm-crb" ,"x-allow-chunk-migration" ,"off" }, + {TYPE_ARM_SMMUV3 ,"ats" ,"off" }, + {TYPE_ARM_SMMUV3 ,"ril" ,"on" }, + {TYPE_ARM_SMMUV3 ,"ssidsize" ,"0" }, + {TYPE_ARM_SMMUV3 ,"oas" ,"44" }, };const size_t hw_compat_11_0_len =G_N_ELEMENTS (hw_compat_11_0);
Signature · 1 line
2.43.0
PATCH v6 9/9 qemu-options.hx: Support "auto" for accel SMMUv3 properties
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Update documentation now that "auto" is supported for accelerated SMMUv3 properties.
--- qemu-options.hx | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
qemu-options.hx +19 -6
@@ -1291,31 +1291,44 @@ SRST Enabling accel configures the host SMMUv3 in nested mode to support vfio-pci passthrough. - The following options are available when accel=on. - Note: 'auto' mode is not currently supported. + The following options will be set to auto by default if not manually + set. When accel=on and these properties are set to auto, the value is + derived from the host SMMUv3 capabilities via IOMMU_GET_HW_INFO. With + accel=on, this requires at least one cold-plugged vfio-pci device; if + none is present at machine init, QEMU will abort. - ``ril=on|off`` (default: on) + If accel=off, auto values resolve to the non-accel defaults given below. + + ``ril=on|off|auto`` (default: auto) Support for Range Invalidation, which allows the SMMUv3 driver to invalidate TLB entries for a range of IOVAs at once instead of issuing separate commands to invalidate each page. Must match with host SMMUv3 Range Invalidation support. - ``ats=on|off`` (default: off) + - With accel=off, auto is resolved to 'on'. + + ``ats=on|off|auto`` (default: auto) Support for Address Translation Services, which enables PCIe devices to cache address translations in their local TLB and reduce latency. Host SMMUv3 must support ATS in order to enable this feature for the vIOMMU. - ``oas=val`` (supported values are 44 and 48. default: 44) + - With accel=off, auto is resolved to 'off'. + + ``oas=val|auto`` (supported values are 44 and 48. default: auto) Sets the Output Address Size in bits. The value set here must be less than or equal to the host SMMUv3's supported OAS, so that the intermediate physical addresses (IPA) consumed by host SMMU for stage-2 translation do not exceed the host's max supported IPA size. - ``ssidsize=val`` (val between 0 and 20. default: 0) + - With accel=off, auto is resolved to 44. + + ``ssidsize=val|auto`` (val between 0 and 20. default: auto) Sets the Substream ID size in bits. When set to a non-zero value, PASID capability is advertised to the vIOMMU and accelerated use cases such as Shared Virtual Addressing (SVA) are supported. + - With accel=off, auto is resolved to 0. + ``-device amd-iommu[,option=...]`` Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU). Only available with ``-machine q35``, it supports the following options:
Signature · 1 line
2.43.0
Hi Nathan, Peter,
Quoted 19 lines · Nathan Chen
Hi,
This is a follow-up to the series [0] that introduces support for resolving 'auto' for arm-smmuv3 accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties based on host IOMMU capabilities. This is dependent on the series [1] for changing these property types to accept 'auto' values.
Accelerated SMMUv3 Address Translation Services support is derived from IDR0, Range Invalidation support is derived from IDR3, Substream ID size is derived from IDR1, and output address space is derived from IDR5.
The default values are set to 'auto' for all properties. If accel=off and the values are set to 'auto' or are omitted and resolve to 'auto', the default property values defined in
smmuv3_init_id_regs()for OAS and RIL will remain unchanged, while SSIDSIZE and ATS values will remain initialized at 0.
Quoted 4 lines
A complete branch can be found here:
https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto-resolve-v6Please take a look and let me know your feedback.
The series looks good to me and I tested it with Grace-Hopper GPU passthrough
with explicit option settings and auto settings. It collected R-bs for all the patches. Feel free to collect my T-b:
Peter, please could you consider pulling it if it suits you.
There is also
[PATCH v2] hw/pci/pci: Enforce pci_setup_iommu_per_bus() is called only once per bus
which collected R-bs and does not clash with the above series
Thanks
Eric
Quoted 72 lines
Thanks,
Nathan[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/qemu-arm/[email protected]/Changes from v5:
- Added R-by tags, thanks all!
- Rebase over latest upstream
(https://lore.kernel.org/all/[email protected]/)Changes from v4:
- Added R-by tags, thanks all!
- Re-wordsmmuv3_class_init()accel description to list nested translation support as a pre-requisite
- Fix typo insmmuv3_class_init()SubstreamID size descriptionChanges from v3:
- Added R-by tags, thanks all!
- Include a fix patch for resolving ATS support in ATC invalidation case
- Update host requirements documented for accel=on, oas, and ssidsize
- Fix style issue, update comments
- Update function name forsmmuv3_accel_machine_done
- Remove unnecessary early return forsmmuv3_accel_machine_done
- Fix formatting in qemu-options.hx based on html renderChanges from v2:
- Added R-by tags, thanks Eric, Shameer, Cédric for the feedback!
- Re-worded descriptions in qemu-options.hx andsmmu_validate_property()
- Emphasize accel=off auto behavior in a comment for smmuv3_properties[]Changes from v1:
- Update qemu-options.hx to explicitly mention auto is supported
- Update qemu-options.hx to mention auto as default
- Simplify comments insmmuv3_accel_auto_finalise()
- Update descriptions insmmuv3_class_init()
- Consolidate all default value change patches into one patch
- Implement and usesmmuv3_pasid_supported()for setting PASID support flag
- Movesmmuv3_machine_done()andqemu_add_machine_init_done_notifier()tosmmuv3-accel.c
- Register machine done notifier only in caseauto_modeis setTesting:
Basic sanity testing was performed on an NVIDIA Grace platform with GPU device assignment and running CUDA test apps on the guest. Observed the feature properties being set based on host IOMMU capabilities. Verified that the VM boot will fail without a cold-plugged device, and that a hot-plugged device re-uses the resolved values from the initial cold-plug. Additional testing and feedback are welcome.Nathan Chen (9):
hw/arm/smmuv3: Update ATC invalidation check hw/arm/smmuv3: Improve accel SMMUv3 usage documentation hw/arm/smmuv3-accel: Add helper for resolving auto parameters hw/arm/smmuv3-accel: Implement "auto" value for "ats" hw/arm/smmuv3-accel: Implement "auto" value for "ril" hw/arm/smmuv3-accel: Implement "auto" value for "ssidsize" hw/arm/smmuv3-accel: Implement "auto" value for "oas" hw/arm/smmuv3: Set default ats, ril, ssidsize, oas to auto qemu-options.hx: Support "auto" for accel SMMUv3 propertieshw/arm/smmuv3-accel.c | 77 ++++++++++++++++++++++++++++++++++++++- hw/arm/smmuv3-accel.h | 2 + hw/arm/smmuv3.c | 81 +++++++++++++++++++++++------------------ hw/core/machine.c | 5 +++ include/hw/arm/smmuv3.h | 3 ++ qemu-options.hx | 25 ++++++++++--- 6 files changed, 150 insertions(+), 43 deletions(-)
Quoted 28 lines · Eric Auger
Hi Nathan, Peter,
Quoted 19 lines · Nathan Chen
Hi,
This is a follow-up to the series [0] that introduces support for resolving 'auto' for arm-smmuv3 accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties based on host IOMMU capabilities. This is dependent on the series [1] for changing these property types to accept 'auto' values.
Accelerated SMMUv3 Address Translation Services support is derived from IDR0, Range Invalidation support is derived from IDR3, Substream ID size is derived from IDR1, and output address space is derived from IDR5.
The default values are set to 'auto' for all properties. If accel=off and the values are set to 'auto' or are omitted and resolve to 'auto', the default property values defined in
smmuv3_init_id_regs()for OAS and RIL will remain unchanged, while SSIDSIZE and ATS values will remain initialized at 0.Quoted 4 lines
A complete branch can be found here:
https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto-resolve-v6Please take a look and let me know your feedback.
Quoted 8 lines
The series looks good to me and I tested it with Grace-Hopper GPU passthrough
with explicit option settings and auto settings. It collected R-bs for all the patches. Feel free to collect my T-b:Tested-by Eric Auger <[email protected]>Peter, please could you consider pulling it if it suits you.
Thanks for taking on the testing and review for this -- I've applied it to target-arm.next.
-- PMM